Systems and methods for operating a head-mounted display system based on user identity

ABSTRACT

Systems and methods for depth plane selection in display system such as augmented reality display systems, including mixed reality display systems, are disclosed. A display(s) may present virtual image content via image light to an eye(s) of a user. The display(s) may output the image light to the eye(s) of the user, the image light to have different amounts of wavefront divergence corresponding to different depth planes at different distances away from the user. A camera(s) may capture images of the eye(s). An indication may be generated based on obtained images of the eye(s), indicating whether the user is identified. The display(s) may be controlled to output the image light to the eye(s) of the user, the image light to have the different amounts of wavefront divergence based at least in part on the generated indication indicating whether the user is identified.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional applicationSer. No. 17/116,999, filed Dec. 9, 2020 which claims priority under 35U.S.C. § 119(e) from U.S. Provisional Application No. 62/945,517, filedon Dec. 9, 2019, entitled “SYSTEMS AND METHODS FOR OPERATING AHEAD-MOUNTED DISPLAY SYSTEM BASED ON USER IDENTITY,” which is herebyincorporated by reference in its entirety.

INCORPORATION BY REFERENCE

This application incorporates by reference the entirety of each of thefollowing patent applications and publications: U.S. application Ser.No. 14/555,585 filed on Nov. 27, 2014, published on Jul. 23, 2015 asU.S. Publication No. 2015/0205126; U.S. application Ser. No. 14/690,401filed on Apr. 18, 2015, published on Oct. 22, 2015 as U.S. PublicationNo. 2015/0302652; U.S. application Ser. No. 14/212,961 filed on Mar. 14,2014, now U.S. Pat. No. 9,417,452 issued on Aug. 16, 2016; U.S.application Ser. No. 14/331,218 filed on Jul. 14, 2014, published onOct. 29, 2015 as U.S. Publication No. 2015/0309263; U.S. applicationSer. No. 15/927,808 filed on Mar. 21, 2018; U.S. application Ser. No.15/291,929 filed on Oct. 12, 2016, published on Apr. 20, 2017 as U.S.Publication No. 2017/0109580; U.S. application Ser. No. 15/408,197 filedon Jan. 17, 2017, published on Jul. 20, 2017 as U.S. Publication No.2017/0206412; U.S. application Ser. No. 15/469,369 filed on Mar. 24,2017, published on Sep. 28, 2017 as U.S. Publication No. 2017/0276948;U.S. Provisional Application No. 62/618,559 filed on Jan. 17, 2018; U.S.application Ser. No. 16/250,931 filed on Jan. 17, 2019; U.S. applicationSer. No. 14/705,741 filed on May 6, 2015, published on Apr. 21, 2016 asU.S. Publication No. 2016/0110920; U.S. patent publication No.2017/0293145, published Oct. 12, 2017; U.S. patent application Ser. No.15/993,371 filed on May 30, 2018, published on Dec. 6, 2018, as U.S.Publication No. 2018/0348861; U.S. patent application Ser. No.15/155,013 filed on May 14, 2016, published on Dec. 8, 2016 as U.S.Publication No. 2016/0358181; U.S. patent application Ser. No.15/934,941 filed on Mar. 23, 2018, published on Sep. 27, 2018 as U.S.Publication No. 2018/0276467; U.S. Provisional Application No.62/644,321 filed on Mar. 16, 2018; U.S. application Ser. No. 16/251,017filed on Jan. 17, 2019; U.S. Provisional Application No. 62/702,866filed on Jul. 24, 2018; International Patent Application No.PCT/US2019/043096 filed on Jul. 23, 2019; U.S. Provisional ApplicationNo. 62/714,649 filed on Aug. 3, 2018; U.S. Provisional Application No.62/875,474 filed on Jul. 17, 2019; and U.S. application Ser. No.16/530,904 filed on Aug. 2, 2019.

FIELD

The present disclosure relates to display systems, virtual reality, andaugmented reality imaging and visualization systems and, moreparticularly, to depth plane selection based in part on a user'sidentity.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time.

The human visual perception system is very complex, and producing a VR,AR, or MR technology that facilitates a comfortable, natural-feeling,rich presentation of virtual image elements amongst other virtual orreal-world imagery elements is challenging. Systems and methodsdisclosed herein address various challenges related to VR, AR and MRtechnology.

SUMMARY

Various examples of systems and methods for depth plane selection indisplay system such as augmented reality display systems, includingmixed reality display systems, are disclosed.

A wearable system may include one or more displays, one or more cameras,and one or more processors. The one or more displays may be configuredto present virtual image content via image light to one or both eyes ofa user, wherein the one or more displays are configured for outputtingthe image light to one or both eyes of the user, the image light to havedifferent amounts of wavefront divergence corresponding to differentdepth planes at different distances away from the user. The one or morecameras may be configured for capturing images of one or both eyes ofthe user. The one or more processors may be configured for: obtainingone or more images of one or both eyes of the user as captured by theone or more cameras; generating an indication based on the one or moreobtained images of one or both eyes of the user, the generatedindication indicating whether the user is identified; and controllingthe one or more displays to output the image light to one or both eyesof the user, the image light to have the different amounts of wavefrontdivergence based at least in part on the generated indication indicatingwhether the user is identified.

In some system embodiments, generating the indication indicating whetherthe user is identified may include performing one or more iris scanningoperations or one or more iris authentication operations.

In some system embodiments, the generated indication indicates that theuser is identified as a registered user and, in response to thegenerated indication indicating that the user is identified as aregistered user, the one or more processors may be configured forperforming the controlling the one or more displays to output imagelight to one or both eyes of the user to include controlling the one ormore displays to output the image light to one or both eyes of the user,the image light to have the different amounts of wavefront divergencebased at least in part on settings associated with the registered user.

In some system embodiments, the generated indication indicates that theuser is not identified as a registered user and, in response to thegenerating indication indicating that the user is not identified as aregistered user, the one or more processors may be configured forperforming the controlling the one or more displays to output the imagelight to one or both eyes of the user to include controlling the one ormore displays to output the image light to one or both eyes of the user,the image light to have the different amounts of wavefront divergencebased at least in part on a set of default settings.

In some system embodiments, the generated indication indicates that theuser is not identified as a registered user and, in response to thegenerating indication indicating that the user is not identified as aregistered user, the one or more processors may be configured for:performing a set of one or more operations to enroll the user in irisauthentication or generate a calibration profile for the user; anddetermining settings for the user based at least in part on informationobtained through performing the set of one or more operations. Thecontrolling the one or more displays to output image light to one orboth eyes of the user may include controlling the one or more displaysto output the image light to one or both eyes of the user, the imagelight to have the different amounts of wavefront divergence based atleast in part on the settings determined for the user.

In some system embodiments, the one or more processors may be configuredfor controlling the one or more displays to present a virtual target tothe user. The obtaining one or more images of one or both eyes of theuser as captured by the one or more cameras may include obtaining theone or more images of one or both eyes of the user as captured by theone or more cameras while the virtual target is being presented to theuser.

In some system embodiments, the one or more processors may be configuredfor performing one or more operations to attempt to improve an accuracyor a reliability of user identification in the generating the indicationindicating whether the user is identified.

A method may include presenting, by one or more displays, virtual imagecontent via image light to one or both eyes of a user, wherein the oneor more displays are configured for outputting the image light to one orboth eyes of the user, the image light to have different amounts ofwavefront divergence corresponding to different depth planes atdifferent distances away from the user. The method may includecapturing, by one or more cameras, images of one or both eyes of theuser; and obtaining one or more images of one or both eyes of the useras captured by the one or more cameras. The method may includegenerating an indication based on the one or more obtained images of oneor both eyes of the user, the generated indication indicating whetherthe user is identified; and controlling the one or more displays tooutput the image light to one or both eyes of the user, the image lightto have the different amounts of wavefront divergence based at least inpart on the generated indication indicating whether the user isidentified.

In some method embodiments, generating the indication indicating whetherthe user is identified may include performing one or more iris scanningoperations or one or more iris authentication operations.

In some method embodiments, the generated indication indicates that theuser is identified as a registered user. The method may include, inresponse to the generated indication indicating that the user isidentified as a registered user, performing the controlling the one ormore displays to output image light to one or both eyes of the user toinclude controlling the one or more displays to output the image lightto one or both eyes of the user, the image light to have the differentamounts of wavefront divergence based at least in part on settingsassociated with the registered user.

In some method embodiments, the generated indication indicates that theuser is not identified as a registered user. The method may include, inresponse to the generating indication indicating that the user is notidentified as a registered user, performing the controlling the one ormore displays to output the image light to one or both eyes of the userto include controlling the one or more displays to output the imagelight to one or both eyes of the user, the image light to have thedifferent amounts of wavefront divergence based at least in part on aset of default settings.

In some method embodiments, wherein the generated indication indicatesthat the user is not identified as a registered user. The method mayinclude, in response to the generating indication indicating that theuser is not identified as a registered user, performing a set of one ormore operations to enroll the user in iris authentication or generate acalibration profile for the user; and determining settings for the userbased at least in part on information obtained through performing theset of one or more operations. The controlling the one or more displaysto output image light to one or both eyes of the user may includecontrolling the one or more displays to output the image light to one orboth eyes of the user, the image light to have the different amounts ofwavefront divergence based at least in part on the settings determinedfor the user.

In some method embodiments, the method may include controlling the oneor more displays to present a virtual target to the user. The obtainingone or more images of one or both eyes of the user as captured by theone or more cameras may include obtaining the one or more images of oneor both eyes of the user as captured by the one or more cameras whilethe virtual target is being presented to the user.

In some method embodiments, the method may include performing one ormore operations to attempt to improve an accuracy or a reliability ofuser identification in the generating the indication indicating whetherthe user is identified.

A non-transitory computer-readable medium may store instructions that,when executed by one or more processors of a wearable system, cause thewearable system to perform a method. The method may include presenting,by one or more displays of the wearable system, virtual image contentvia image light to one or both eyes of a user, wherein the one or moredisplays are configured for outputting the image light to one or botheyes of the user, the image light to have different amounts of wavefrontdivergence corresponding to different depth planes at differentdistances away from the user. The method may include capturing, by oneor more cameras of the wearable system, images of one or both eyes ofthe user; and obtaining one or more images of one or both eyes of theuser as captured by the one or more cameras. The method may includegenerating an indication based on the one or more obtained images of oneor both eyes of the user, the generated indication indicating whetherthe user is identified; and controlling the one or more displays tooutput the image light to one or both eyes of the user, the image lightto have the different amounts of wavefront divergence based at least inpart on the generated indication indicating whether the user isidentified.

In some non-transitory computer-readable medium embodiments, generatingthe indication indicating whether the user is identified may includeperforming one or more iris scanning operations or one or more irisauthentication operations.

In some non-transitory computer-readable medium embodiments, thegenerated indication indicates that the user is identified as aregistered user. The method may include, in response to the generatedindication indicating that the user is identified as a registered user,performing the controlling the one or more displays to output imagelight to one or both eyes of the user to include controlling the one ormore displays to output the image light to one or both eyes of the user,the image light to have the different amounts of wavefront divergencebased at least in part on settings associated with the registered user.

In some non-transitory computer-readable medium embodiments, thegenerated indication indicates that the user is not identified as aregistered user. The method may include, in response to the generatingindication indicating that the user is not identified as a registereduser, performing the controlling the one or more displays to output theimage light to one or both eyes of the user to include controlling theone or more displays to output the image light to one or both eyes ofthe user, the image light to have the different amounts of wavefrontdivergence based at least in part on a set of default settings.

In some non-transitory computer-readable medium embodiments, thegenerated indication indicates that the user is not identified as aregistered user. The method may include, in response to the generatingindication indicating that the user is not identified as a registereduser, performing a set of one or more operations to enroll the user iniris authentication or generate a calibration profile for the user; anddetermining settings for the user based at least in part on informationobtained through performing the set of one or more operations. Thecontrolling the one or more displays to output image light to one orboth eyes of the user may include controlling the one or more displaysto output the image light to one or both eyes of the user, the imagelight to have the different amounts of wavefront divergence based atleast in part on the settings determined for the user.

In some non-transitory computer-readable medium embodiments, the methodmay include controlling the one or more displays to present a virtualtarget to the user. The obtaining one or more images of one or both eyesof the user as captured by the one or more cameras may include obtainingthe one or more images of one or both eyes of the user as captured bythe one or more cameras while the virtual target is being presented tothe user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates example components of a wearablesystem.

FIG. 4 schematically illustrates an example of a waveguide stack of awearable device for outputting image information to a user.

FIG. 5 schematically illustrates an example of an eye, as well as anexample coordinate system for determining an eye pose of an eye.

FIG. 6 is a schematic diagram of a wearable system that includes an eyetracking system.

FIG. 7A is a block diagram of a wearable system that may include an eyetracking system.

FIG. 7B is a block diagram of a render controller in a wearable system.

FIG. 7C is a block diagram of a registration observer in a head-mounteddisplay system.

FIG. 8A is a schematic diagram of an eye showing the eye's cornealsphere.

FIG. 8B illustrates an example corneal glint detected by an eye trackingcamera.

FIGS. 8C-8E illustrate example stages of locating a user's cornealcenter with an eye tracking module in a wearable system.

FIGS. 9A-9C illustrate an example normalization of the coordinate systemof eye tracking images.

FIGS. 9D-9G illustrate example stages of locating a user's pupil centerwith an eye tracking module in a wearable system.

FIG. 10 illustrates an example of an eye including the eye's optical andvisual axes and the eye's center of rotation.

FIG. 11 is a process flow diagram of an example of a method for usingeye tracking in rendering content and providing feedback on registrationin a wearable device.

FIGS. 12A and 12B illustrate a nominal position of a display elementrelative to a user's eye and illustrate a coordinate system fordescribing the positions of the display element and the user's eyerelative to one another.

FIG. 13 is a set of example graphs illustrating how the wearable systemmay switch depth planes in response to the user's eye movements.

FIG. 14 is a process flow diagram of an example of a method for depthplane selection using an existing calibration, a dynamic calibration,and/or a content-based switching scheme.

FIG. 15 is a process flow diagram of an example of a method for depthplane selection based at least partly on a user's interpupillarydistance.

FIG. 16A illustrates an example of a top-down view of a representationof a user viewing content presented by a display system configured toswitch depth planes by detecting user fixation within one of a pluralityof zones that segment the user's field of view along the horizontalaxis.

FIG. 16B illustrates an example of a perspective view of therepresentation of FIG. 16A.

FIG. 17A illustrates an example of a top-down view of a representationof a user viewing content presented by a display system configured toswitch depth planes by detecting user fixation within discrete markervolumes within the display frustum.

FIG. 17B illustrates an example of a perspective view of therepresentation of FIG. 16A.

FIG. 18 illustrates a flowchart of an example process to select a depthplane based on content-based switching.

FIG. 19 illustrates a flowchart of another example process to adjustzones based on content-based switching.

FIG. 20 illustrates a flowchart of an example process for operating ahead-mounted display system based on user identity.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

As described herein, display systems (e.g., augmented reality or virtualreality display systems) may render virtual content for presentation toa user at different perceived depths from the user. In augmented realitydisplay systems, different depth planes may be utilized to projectvirtual content with each depth plane being associated with a particularperceived depth from the user. For example, a stack of waveguidesconfigured to output light with different wavefront divergences may beutilized, with each depth plane having a corresponding wavefrontdivergence and being associated with at least one waveguide. As virtualcontent moves about the user's field of view, the virtual content may beadjusted along three discrete axes. For example, the virtual content maybe adjusted along the X, Y, and Z axes such that the virtual content maybe presented at different perceived depths from the user. The displaysystem may switch between depth planes as the virtual content isperceived to be moved further from, or closer to, the user. It will beappreciated that switching depth planes may involve changing thewavefront divergence of light forming the virtual content in a discretestep. In a waveguide-based system, in some embodiments, such a depthplane switch may involve switching the waveguide outputting light toform the virtual content.

In some embodiments, the display system may be configured to monitor thegaze of a user's eyes and determine a three-dimensional fixation pointat which the user is fixating. The fixation point may be determinedbased upon, among other things, the distance between the user's eyes andthe gaze direction of each eye. It will be appreciated that thesevariables may be understood to form a triangle with the fixation pointat one corner of the triangle and the eyes at the other corners. It willalso be appreciated that a calibration may be performed to accuratelytrack the orientations of the user's eyes and determine or estimate thegaze of those eyes to determine the fixation point. Thus, afterundergoing a full calibration, the display device may have a calibrationfile or calibration information for the main user of that device. Themain user may also be referred to as a calibrated user herein. Furtherdetails regarding calibration and eye tracking may be found in, e.g.,U.S. patent application Ser. No. 15/993,371, entitled “EYE TRACKINGCALIBRATION TECHNIQUES,” which is incorporated herein by reference inits entirety.

The display system may occasionally be used by guest users that have notcompleted the full calibration. In addition, these guest users may nothave the time or the desire to perform a full calibration. However, ifthe display system does not actually track the fixation point of theguest user, then the depth plane switching may not be appropriate forproviding a realistic and comfortable viewing experience for the user.

Thus, it will be appreciated that the current user of the display systemmay be categorized as a calibrated user or as a guest user. In someembodiments, the display system may be configured to categorize thecurrent user by determining whether or not the current user is acalibrated user by performing an identification, or authentication,process, e.g., to determine whether information provided by and/orobtained from the current user matches information associated with thecalibrated user. For example, the authentication or identificationprocess may be one or more of: asking for and verifying a usernameand/or password, conducting iris scanning (e.g., by comparing a currentimage of the user's iris with a reference image), conducting voicerecognition (e.g., by comparing a current sample of the user's voicewith a reference voice file), and IPD matching. In some embodiments, IPDmatching may include determining whether the current user's IPD matchesthe calibrated user's IPD. If there is a match, the current user may beassumed to be the calibrated user in some embodiments. If there is nomatch, the current user may be assumed to not be the calibrated user(e.g., to be a guest user) in some embodiments. In some otherembodiments, multiple authentication processes may be conducted for acurrent user to increase the accuracy of the determination of whetherthe current user is a calibrated user.

In some embodiments, if the current user is determined to not be acalibrated user (e.g., a guest user), the display system may usecontent-based depth plane switching; for example, rather thandetermining the fixation point of the user's eyes and switch depthplanes based on the depth plane in which the fixation point is location,the display system may be configured to display content with theappropriate amount of wavefront divergence for where, in 3D space, thatcontent (e.g., a virtual object) is specified to be placed. It will beappreciated that virtual objects may have associated locations orcoordinates in the three-dimensional volume around the user, and thedisplay system may be configured to present the object using light withthe amount of wavefront divergence appropriate for the object's depth,relative to the user, within that three-dimensional volume.

In cases where multiple virtual objects at different depths are to bedisplayed, content-based depth plane switching may involve making arough determination as to which virtual object is being fixated on andthen using the location of that virtual object to establish the plane towhich depth plane switching should switch; for example, upon determiningthat the user is roughly fixating on a particular virtual object, thedisplay system may be configured to output light with wavefrontdivergence corresponding to the depth plane associated with that virtualobject. In some embodiments, this rough determination of whether theuser is fixating on an object may involve determining whether thefixation point is within a display system-defined volume unique to theobject and, if yes, switching to the depth plane associated with thatobject, irrespective of the depth of the determined fixation point (sothat, if the volume extends across multiple depth planes, the displaysystem will switch to the depth plane associated with the object).

In some embodiments, if the current user is determined to not be acalibrated user (e.g., a guest user) the display system may perform arough calibration by measuring the guest user's interpupillary distance(IPD). This rough calibration may also be referred to as a dynamiccalibration. Preferably, the IPD is measured while the user's eyes aredirected to, or focused on an object at, optical infinity. In someembodiments, this IPD value may be understood to be the maximum IPDvalue. The maximum IPD value may be used or a lesser value selectedwithin a distribution of sampled values (e.g., the value at the 95^(th)percentile of sampled IPD values) may be utilized as a reference valuefor determining the fixation point. For example, this IPD value mayconstitute one side (e.g., the base) of the imaginary triangle of whichthe fixation point forms a corner (e.g., the apex).

Thus, in some embodiments, the display system may be configured tomonitor for whether the user is a main or guest user. If the user is amain user, then the calibration file may be accessed. If the user is aguest user, then content-based depth plane switching may be utilized,and/or a rough calibration involving determining IPD may be performed toestablish a reference IPD value. The display system may be configured touse this reference IPD value to determine or estimate the guest user'sfixation point and, thus, make decisions regarding when to switch depthplanes (e.g. when to switch the wavefront divergence of light used toform virtual content).

In some embodiments, the display system may transition from performingcontent-based depth plane switching to performing depth plane switchingbased on a dynamic calibration of the current user. For example, such atransition may occur in situations where data obtained in connectionwith the content-based depth plane switching scheme is deemed to beunreliable (e.g., where values for the virtual object that the user isroughly fixating on have high levels of uncertainty or variability), orwhere content is provided at a range of different depths spanningmultiple depth planes (e.g., wherein the virtual content spans greaterthan a threshold number of depth planes).

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout. Unless indicated otherwise, thedrawings are schematic not and necessarily drawn to scale.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. At least a portion of the wearable system canbe implemented on a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction. The wearabledevice can be used interchangeably as an AR device (ARD). Further, forthe purpose of the present disclosure, the term “AR” is usedinterchangeably with the term “MW”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1 , an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2 illustrates an example of wearable system 200 which can beconfigured to provide an AR/VR/MR scene. The wearable system 200 canalso be referred to as the AR system 200. The wearable system 200includes a display 220, and various mechanical and electronic modulesand systems to support the functioning of display 220. The display 220may be coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 can be positioned in front of the eyes ofthe user 210. The display 220 can present AR/VR/MR content to a user.The display 220 can comprise a head mounted display (HMD) that is wornon the head of the user.

In some embodiments, a speaker 240 is coupled to the frame 230 andpositioned adjacent the ear canal of the user (in some embodiments,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). The display220 can include an audio sensor (e.g., a microphone) 232 for detectingan audio stream from the environment and capture ambient sound. In someembodiments, one or more other audio sensors, not shown, are positionedto provide stereo sound reception. Stereo sound reception can be used todetermine the location of a sound source. The wearable system 200 canperform voice or speech recognition on the audio stream.

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4 ) which observes the world in the environment aroundthe user. The wearable system 200 can also include an inward-facingimaging system 462 (shown in FIG. 4 ) which can track the eye movementsof the user. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210. The inward-facing imaging system 462 mayinclude one or more cameras. For example, at least one camera may beused to image each eye. The images acquired by the cameras may be usedto determine pupil size or eye pose for each eye separately, therebyallowing presentation of image information to each eye to be dynamicallytailored to that eye.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; orb)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules are availableas resources to the local processing and data module 260. In addition,remote processing module 280 and remote data repository 280 may beoperatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

Example Components of a Wearable System

FIG. 3 schematically illustrates example components of a wearablesystem. FIG. 3 shows a wearable system 200 which can include a display220 and a frame 230. A blown-up view 202 schematically illustratesvarious components of the wearable system 200. In certain implements,one or more of the components illustrated in FIG. 3 can be part of thedisplay 220. The various components alone or in combination can collecta variety of data (such as e.g., audio or visual data) associated withthe user of the wearable system 200 or the user's environment. It shouldbe appreciated that other embodiments may have additional or fewercomponents depending on the application for which the wearable system isused. Nevertheless, FIG. 3 provides a basic idea of some of the variouscomponents and types of data that may be collected, analyzed, and storedthrough the wearable system.

FIG. 3 shows an example wearable system 200 which can include thedisplay 220. The display 220 can comprise a display lens 226 that may bemounted to a user's head or a housing or frame 230, which corresponds tothe frame 230. The display lens 226 may comprise one or more transparentmirrors positioned by the housing 230 in front of the user's eyes 302,304 and may be configured to bounce projected light 338 into the eyes302, 304 and facilitate beam shaping, while also allowing fortransmission of at least some light from the local environment. Thewavefront of the projected light beam 338 may be bent or focused tocoincide with a desired focal distance of the projected light. Asillustrated, two wide-field-of-view machine vision cameras 316 (alsoreferred to as world cameras) can be coupled to the housing 230 to imagethe environment around the user. These cameras 316 can be dual capturevisible light/non-visible (e.g., infrared) light cameras. The cameras316 may be part of the outward-facing imaging system 464 shown in FIG. 4. Image acquired by the world cameras 316 can be processed by the poseprocessor 336. For example, the pose processor 336 can implement one ormore object recognizers 708 (e.g., shown in FIG. 7 ) to identify a poseof a user or another person in the user's environment or to identify aphysical object in the user's environment.

With continued reference to FIG. 3 , a pair of scanned-lasershaped-wavefront (e.g., for depth) light projector modules with displaymirrors and optics configured to project light 338 into the eyes 302,304 are shown. The depicted view also shows two miniature infraredcameras 324 paired with infrared light sources 326 (such as lightemitting diodes “LED”s), which are configured to be able to track theeyes 302, 304 of the user to support rendering and user input. Thecameras 324 may be part of the inward-facing imaging system 462 shown inFIG. 4 . The wearable system 200 can further feature a sensor assembly339, which may comprise X, Y, and Z axis accelerometer capability aswell as a magnetic compass and X, Y, and Z axis gyro capability,preferably providing data at a relatively high frequency, such as 200Hz. The sensor assembly 339 may be part of the IMU described withreference to FIG. 2A The depicted system 200 can also comprise a headpose processor 336, such as an ASIC (application specific integratedcircuit), FPGA (field programmable gate array), or ARM processor(advanced reduced-instruction-set machine), which may be configured tocalculate real or near-real time user head pose from wide field of viewimage information output from the capture devices 316. The head poseprocessor 336 can be a hardware processor and can be implemented as partof the local processing and data module 260 shown in FIG. 2A.

The wearable system can also include one or more depth sensors 234. Thedepth sensor 234 can be configured to measure the distance between anobject in an environment to a wearable device. The depth sensor 234 mayinclude a laser scanner (e.g., a lidar), an ultrasonic depth sensor, ora depth sensing camera. In certain implementations, where the cameras316 have depth sensing ability, the cameras 316 may also be consideredas depth sensors 234.

Also shown is a processor 332 configured to execute digital or analogprocessing to derive pose from the gyro, compass, or accelerometer datafrom the sensor assembly 339. The processor 332 may be part of the localprocessing and data module 260 shown in FIG. 2 . The wearable system 200as shown in FIG. 3 can also include a position system such as, e.g., aGPS 337 (global positioning system) to assist with pose and positioninganalyses. In addition, the GPS may further provide remotely-based (e.g.,cloud-based) information about the user's environment. This informationmay be used for recognizing objects or information in user'senvironment.

The wearable system may combine data acquired by the GPS 337 and aremote computing system (such as, e.g., the remote processing module270, another user's ARD, etc.) which can provide more information aboutthe user's environment. As one example, the wearable system candetermine the user's location based on GPS data and retrieve a world map(e.g., by communicating with a remote processing module 270) includingvirtual objects associated with the user's location. As another example,the wearable system 200 can monitor the environment using the worldcameras 316 (which may be part of the outward-facing imaging system 464shown in FIG. 4 ). Based on the images acquired by the world cameras316, the wearable system 200 can detect objects in the environment(e.g., by using one or more object recognizers 708 shown in FIG. 7 ).The wearable system can further use data acquired by the GPS 337 tointerpret the characters.

The wearable system 200 may also comprise a rendering engine 334 whichcan be configured to provide rendering information that is local to theuser to facilitate operation of the scanners and imaging into the eyesof the user, for the user's view of the world. The rendering engine 334may be implemented by a hardware processor (such as, e.g., a centralprocessing unit or a graphics processing unit). In some embodiments, therendering engine is part of the local processing and data module 260.The rendering engine 334 can be communicatively coupled (e.g., via wiredor wireless links) to other components of the wearable system 200. Forexample, the rendering engine 334, can be coupled to the eye cameras 324via communication link 274, and be coupled to a projecting subsystem 318(which can project light into user's eyes 302, 304 via a scanned laserarrangement in a manner similar to a retinal scanning display) via thecommunication link 272. The rendering engine 334 can also be incommunication with other processing units such as, e.g., the sensor poseprocessor 332 and the image pose processor 336 via links 276 and 294respectively.

The cameras 324 (e.g., mini infrared cameras) may be utilized to trackthe eye pose to support rendering and user input. Some example eye posesmay include where the user is looking or at what depth he or she isfocusing (which may be estimated with eye vergence). The GPS 337, gyros,compass, and accelerometers 339 may be utilized to provide coarse orfast pose estimates. One or more of the cameras 316 can acquire imagesand pose, which in conjunction with data from an associated cloudcomputing resource, may be utilized to map the local environment andshare user views with others.

The example components depicted in FIG. 3 are for illustration purposesonly. Multiple sensors and other functional modules are shown togetherfor ease of illustration and description. Some embodiments may includeonly one or a subset of these sensors or modules. Further, the locationsof these components are not limited to the positions depicted in FIG. 3. Some components may be mounted to or housed within other components,such as a belt-mounted component, a hand-held component, or a helmetcomponent. As one example, the image pose processor 336, sensor poseprocessor 332, and rendering engine 334 may be positioned in a beltpackand configured to communicate with other components of the wearablesystem via wireless communication, such as ultra-wideband, Wi-Fi,Bluetooth, etc., or via wired communication. The depicted housing 230preferably is head-mountable and wearable by the user. However, somecomponents of the wearable system 200 may be worn to other portions ofthe user's body. For example, the speaker 240 may be inserted into theears of a user to provide sound to the user.

Regarding the projection of light 338 into the eyes 302, 304 of theuser, in some embodiment, the cameras 324 may be utilized to measurewhere the centers of a user's eyes are geometrically verged to, which,in general, coincides with a position of focus, or “depth of focus”, ofthe eyes. A 3-dimensional surface of all points the eyes verge to can bereferred to as the “horopter”. The focal distance may take on a finitenumber of depths, or may be infinitely varying. Light projected from thevergence distance appears to be focused to the subject eye 302, 304,while light in front of or behind the vergence distance is blurred.Examples of wearable devices and other display systems of the presentdisclosure are also described in U.S. Patent Publication No.2016/0270656, which is incorporated by reference herein in its entirety.

The human visual system is complicated and providing a realisticperception of depth is challenging. Viewers of an object may perceivethe object as being three-dimensional due to a combination of vergenceand accommodation. Vergence movements (e.g., rolling movements of thepupils toward or away from each other to converge the lines of sight ofthe eyes to fixate upon an object) of the two eyes relative to eachother are closely associated with focusing (or “accommodation”) of thelenses of the eyes. Under normal conditions, changing the focus of thelenses of the eyes, or accommodating the eyes, to change focus from oneobject to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex.” Likewise, achange in vergence will trigger a matching change in accommodation,under normal conditions. Display systems that provide a better matchbetween accommodation and vergence may form more realistic andcomfortable simulations of three-dimensional imagery.

Further spatially coherent light with a beam diameter of less than about0.7 millimeters can be correctly resolved by the human eye regardless ofwhere the eye focuses. Thus, to create an illusion of proper focaldepth, the eye vergence may be tracked with the cameras 324, and therendering engine 334 and projection subsystem 318 may be utilized torender all objects on or close to the horopter in focus, and all otherobjects at varying degrees of defocus (e.g., using intentionally-createdblurring). Preferably, the system 220 renders to the user at a framerate of about 60 frames per second or greater. As described above,preferably, the cameras 324 may be utilized for eye tracking, andsoftware may be configured to pick up not only vergence geometry butalso focus location cues to serve as user inputs. Preferably, such adisplay system is configured with brightness and contrast suitable forday or night use.

In some embodiments, the display system preferably has latency of lessthan about 20 milliseconds for visual object alignment, less than about0.1 degree of angular alignment, and about 1 arc minute of resolution,which, without being limited by theory, is believed to be approximatelythe limit of the human eye. The display system 220 may be integratedwith a localization system, which may involve GPS elements, opticaltracking, compass, accelerometers, or other data sources, to assist withposition and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (e.g., such information would facilitate the glasses toknow where they are with respect to the real world).

In some embodiments, the wearable system 200 is configured to displayone or more virtual images based on the accommodation of the user'seyes. Unlike prior 3D display approaches that force the user to focuswhere the images are being projected, in some embodiments, the wearablesystem is configured to automatically vary the focus of projectedvirtual content to allow for a more comfortable viewing of one or moreimages presented to the user. For example, if the user's eyes have acurrent focus of 1 m, the image may be projected to coincide with theuser's focus. If the user shifts focus to 3 m, the image is projected tocoincide with the new focus. Thus, rather than forcing the user to apredetermined focus, the wearable system 200 of some embodiments allowsthe user's eye to a function in a more natural manner.

Such a wearable system 200 may eliminate or reduce the incidences of eyestrain, headaches, and other physiological symptoms typically observedwith respect to virtual reality devices. To achieve this, variousembodiments of the wearable system 200 are configured to project virtualimages at varying focal distances, through one or more variable focuselements (VFEs). In one or more embodiments, 3D perception may beachieved through a multi-plane focus system that projects images atfixed focal planes away from the user. Other embodiments employ variableplane focus, wherein the focal plane is moved back and forth in thez-direction to coincide with the user's present state of focus.

In both the multi-plane focus systems and variable plane focus systems,wearable system 200 may employ eye tracking to determine a vergence ofthe user's eyes, determine the user's current focus, and project thevirtual image at the determined focus. In other embodiments, wearablesystem 200 comprises a light modulator that variably projects, through afiber scanner, or other light generating source, light beams of varyingfocus in a raster pattern across the retina. Thus, the ability of thedisplay of the wearable system 200 to project images at varying focaldistances not only eases accommodation for the user to view objects in3D, but may also be used to compensate for user ocular anomalies, asfurther described in U.S. Patent Publication No. 2016/0270656, which isincorporated by reference herein in its entirety. In some otherembodiments, a spatial light modulator may project the images to theuser through various optical components. For example, as describedfurther below, the spatial light modulator may project the images ontoone or more waveguides, which then transmit the images to the user.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 440 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In some embodiments, the lens may be variable focus elements(VFEs). For example, in some embodiments, the waveguide assembly 480 maysimply include two variable focus elements and one or more waveguides inbetween those two variable focus elements. Examples of waveguideassemblies with VFEs are disclosed in US patent publication No.2017/0293145, published Oct. 12, 2017, the entire disclosure of which isincorporated herein by reference. In other embodiments, the features458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers(e.g., cladding layers or structures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2 ) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4 , as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. (Compensating lens layer 430 and thestacked waveguide assembly 480 as a whole may be configured such thatlight coming from the world 470 is conveyed to the eye 410 atsubstantially the same level of divergence (or collimation) as the lighthad when it was initially received by the stacked waveguide assembly480.) Such a configuration provides as many perceived focal planes asthere are available waveguide/lens pairings. Both the light extractingoptical elements of the waveguides and the focusing aspects of thelenses may be static (e.g., not dynamic or electro-active). In somealternative embodiments, either or both may be dynamic usingelectro-active features.

With continued reference to FIG. 4 , the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) can beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer can move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 can be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 can include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors can be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 can comprise a directionalmicrophone, as another example, which can also provide such usefuldirectional information as to where the audio source is located. Thewearable system 400 can use information from both the outward-facingimaging system 464 and the audio sensor 230 in locating a source ofspeech, or to determine an active speaker at a particular moment intime, etc. For example, the wearable system 400 can use the voicerecognition alone or in combination with a reflected image of thespeaker (e.g., as seen in a mirror) to determine the identity of thespeaker. As another example, the wearable system 400 can determine aposition of the speaker in an environment based on sound acquired fromdirectional microphones. The wearable system 400 can parse the soundcoming from the speaker's position with speech recognition algorithms todetermine the content of the speech and use voice recognition techniquesto determine the identity (e.g., name or other demographic information)of the speaker.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4 ) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

Example of an Eye Image

FIG. 5 illustrates an image of an eye 500 with eyelids 504, sclera 508(the “white” of the eye), iris 512, and pupil 516. Curve 516 a shows thepupillary boundary between the pupil 516 and the iris 512, and curve 512a shows the limbic boundary between the iris 512 and the sclera 508. Theeyelids 504 include an upper eyelid 504 a and a lower eyelid 504 b. Theeye 500 is illustrated in a natural resting pose (e.g., in which theuser's face and gaze are both oriented as they would be toward a distantobject directly ahead of the user). The natural resting pose of the eye500 can be indicated by a natural resting direction 520, which is adirection orthogonal to the surface of the eye 500 when in the naturalresting pose (e.g., directly out of the plane for the eye 500 shown inFIG. 5 ) and in this example, centered within the pupil 516.

As the eye 500 moves to look toward different objects, the eye pose willchange relative to the natural resting direction 520. The current eyepose can be determined with reference to an eye pose direction 524,which is a direction orthogonal to the surface of the eye (and centeredin within the pupil 516) but oriented toward the object at which the eyeis currently directed. With reference to an example coordinate systemshown in FIG. 5A, the pose of the eye 500 can be expressed as twoangular parameters indicating an azimuthal deflection and a zenithaldeflection of the eye pose direction 524 of the eye, both relative tothe natural resting direction 520 of the eye. For purposes ofillustration, these angular parameters can be represented as θ(azimuthal deflection, determined from a fiducial azimuth) and ϕ(zenithal deflection, sometimes also referred to as a polar deflection).In some implementations, angular roll of the eye around the eye posedirection 524 can be included in the determination of eye pose, andangular roll can be included in the following analysis. In otherimplementations, other techniques for determining the eye pose can beused, for example, a pitch, yaw, and optionally roll system.

An eye image can be obtained from a video using any appropriate process,for example, using a video processing algorithm that can extract animage from one or more sequential frames. The pose of the eye can bedetermined from the eye image using a variety of eye-trackingtechniques. For example, an eye pose can be determined by consideringthe lensing effects of the cornea on light sources that are provided.Any suitable eye tracking technique can be used for determining eye posein the eyelid shape estimation techniques described herein.

Example of an Eye Tracking System

FIG. 6 illustrates a schematic diagram of a wearable system 600 thatincludes an eye tracking system. The wearable system 600 may, in atleast some embodiments, include components located in a head-mountedunit 602 and components located in a non-head-mounted unit 604. Non-headmounted unit 604 may be, as examples, a belt-mounted component, ahand-held component, a component in a backpack, a remote component, etc.Incorporating some of the components of the wearable system 600 innon-head-mounted unit 604 may help to reduce the size, weight,complexity, and cost of the head-mounted unit 602. In someimplementations, some or all of the functionality described as beingperformed by one or more components of head-mounted unit 602 and/ornon-head mounted 604 may be provided by way of one or more componentsincluded elsewhere in the wearable system 600. For example, some or allof the functionality described below in association with a CPU 612 ofhead-mounted unit 602 may be provided by way of a CPU 616 of non-headmounted unit 604, and vice versa. In some examples, some or all of suchfunctionality may be provided by way of peripheral devices of wearablesystem 600. Furthermore, in some implementations, some or all of suchfunctionality may be provided by way of one or more cloud computingdevices or other remotely-located computing devices in a manner similarto that which has been described above with reference to FIG. 2 .

As shown in FIG. 6 , wearable system 600 can include an eye trackingsystem including a camera 324 that captures images of a user's eye 610.If desired, the eye tracking system may also include light sources 326 aand 326 b (such as light emitting diodes “LED”s). The light sources 326a and 326 b may generate glints (i.e., reflections off of the user'seyes that appear in images of the eye captured by camera 324). Thepositions of the light sources 326 a and 326 b relative to the camera324 may be known and, as a consequence, the positions of the glintswithin images captured by camera 324 may be used in tracking the user'seyes (as will be discussed in more detail below in connection with FIGS.7-11 ). In at least one embodiment, there may be one light source 326and one camera 324 associated with a single one of the user's eyes 610.In another embodiment, there may be one light source 326 and one camera324 associated with each of a user's eyes. 610. In yet otherembodiments, there may be one or more cameras 324 and one or more lightsources 326 associated with one or each of a user's eyes 610. As aspecific example, there may be two light sources 326 a and 326 b and oneor more cameras 324 associated with each of a user's eyes 610. Asanother example, there may be three or more light sources such as lightsources 326 a and 326 b and one or more cameras 324 associated with eachof a user's eyes 610.

Eye tracking module 614 may receive images from eye tracking camera(s)324 and may analyze the images to extract various pieces of information.As examples, the eye tracking module 614 may detect the user's eyeposes, a three-dimensional position of the user's eye relative to theeye tracking camera 324 (and to the head-mounted unit 602), thedirection one or both of the user's eyes 610 are focused on, the user'svergence depth (i.e., the depth from the user at which the user isfocusing on), the positions of the user's pupils, the positions of theuser's cornea and cornea sphere, the center of rotation of each of theuser's eyes, and the center of perspective of each of the user's eyes.The eye tracking module 614 may extract such information usingtechniques described below in connection with FIGS. 7-11 . As shown inFIG. 6 , eye tracking module 614 may be a software module implementedusing a CPU 612 in a head-mounted unit 602. Further details discussingthe creation, adjustment, and use of eye tracking module components areprovided in U.S. patent application Ser. No. 15/993,371, entitled “EYETRACKING CALIBRATION TECHNIQUES,” which is incorporated herein byreference in its entirety.

Data from eye tracking module 614 may be provided to other components inthe wearable system. As example, such data may be transmitted tocomponents in a non-head-mounted unit 604 such as CPU 616 includingsoftware modules for a light-field render controller 618 and aregistration observer 620.

Render controller 618 may use information from eye tracking module 614to adjust images displayed to the user by render engine 622 (e.g., arender engine that may be a software module in GPU 620 and that mayprovide images to display 220). As an example, the render controller 618may adjust images displayed to the user based on the user's center ofrotation or center of perspective. In particular, the render controller618 may use information on the user's center of perspective to simulatea render camera (i.e., to simulate collecting images from the user'sperspective) and may adjust images displayed to the user based on thesimulated render camera.

A “render camera,” which is sometimes also referred to as a “pinholeperspective camera” (or simply “perspective camera”) or “virtual pinholecamera” (or simply “virtual camera”), is a simulated camera for use inrendering virtual image content possibly from a database of objects in avirtual world. The objects may have locations and orientations relativeto the user or wearer and possibly relative to real objects in theenvironment surrounding the user or wearer. In other words, the rendercamera may represent a perspective within render space from which theuser or wearer is to view 3D virtual contents of the render space (e.g.,virtual objects). The render camera may be managed by a render engine torender virtual images based on the database of virtual objects to bepresented to the eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a pinhole camera (corresponding to the“render camera”) having a specific set of intrinsic parameters (e.g.,focal length, camera pixel size, principal point coordinates,skew/distortion parameters, etc.), and a specific set of extrinsicparameters (e.g., translational components and rotational componentsrelative to the virtual world). The virtual images are taken from theperspective of such a camera having a position and orientation of therender camera (e.g., extrinsic parameters of the render camera). Itfollows that the system may define and/or adjust intrinsic and extrinsicrender camera parameters. For example, the system may define aparticular set of extrinsic render camera parameters such that virtualimages may be rendered as if captured from the perspective of a camerahaving a specific location with respect to the user's or wearer's eye soas to provide images that appear to be from the perspective of the useror wearer. The system may later dynamically adjust extrinsic rendercamera parameters on-the-fly so as to maintain registration with thespecific location. Similarly, intrinsic render camera parameters may bedefined and dynamically adjusted over time. In some implementations, theimages are rendered as if captured from the perspective of a camerahaving an aperture (e.g., pinhole) at a specific location with respectto the user's or wearer's eye (such as the center of perspective orcenter of rotation, or elsewhere).

In some embodiments, the system may create or dynamically repositionand/or reorient one render camera for the user's left eye, and anotherrender camera for the user's right eye, as the user's eyes arephysically separated from one another and thus consistently positionedat different locations. It follows that, in at least someimplementations, virtual content rendered from the perspective of arender camera associated with the viewer's left eye may be presented tothe user through an eyepiece on the left side of a head-mounted display(e.g., head-mounted unit 602), and that virtual content rendered fromthe perspective of a render camera associated with the user's right eyemay be presented to the user through an eyepiece on the right side ofsuch a head-mounted display. Further details discussing the creation,adjustment, and use of render cameras in rendering processes areprovided in U.S. patent application Ser. No. 15/274,823, entitled“METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN3D RECONSTRUCTION,” which is expressly incorporated herein by referencein its entirety for all purposes.

In some examples, one or more modules (or components) of the system 600(e.g., light-field render controller 618, render engine 620, etc.) maydetermine the position and orientation of the render camera withinrender space based on the position and orientation of the user's headand eyes (e.g., as determined based on head pose and eye tracking data,respectively). For example, the system 600 may effectively map theposition and orientation of the user's head and eyes to particularlocations and angular positions within a 3D virtual environment, placeand orient render cameras at the particular locations and angularpositions within the 3D virtual environment, and render virtual contentfor the user as it would be captured by the render camera. Furtherdetails discussing real world to virtual world mapping processes areprovided in U.S. patent application Ser. No. 15/296,869, entitled“SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE,” which isexpressly incorporated herein by reference in its entirety for allpurposes. As an example, the render controller 618 may adjust the depthsat which images are displayed by selecting which depth plane (or depthplanes) are utilized at any given time to display the images. In someimplementations, such a depth plane switch may be carried out through anadjustment of one or more intrinsic render camera parameters.

Registration observer 620 may use information from eye tracking module614 to identify whether the head-mounted unit 602 is properly positionedon a user's head. As an example, the eye tracking module 614 may provideeye location information, such as the positions of the centers ofrotation of the user's eyes, indicative of the three-dimensionalposition of the user's eyes relative to camera 324 and head-mounted unit602 and the eye tracking module 614 may use the location information todetermine if display 220 is properly aligned in the user's field ofview, or if the head-mounted unit 602 (or headset) has slipped or isotherwise misaligned with the user's eyes. As examples, the registrationobserver 620 may be able to determine if the head-mounted unit 602 hasslipped down the user's nose bridge, thus moving display 220 away anddown from the user's eyes (which may be undesirable), if thehead-mounted unit 602 has been moved up the user's nose bridge, thusmoving display 220 closer and up from the user's eyes, if thehead-mounted unit 602 has been shifted left or right relative the user'snose bridge, if the head-mounted unit 602 has been lifted above theuser's nose bridge, or if the head-mounted unit 602 has been moved inthese or other ways away from a desired position or range of positions.In general, registration observer 620 may be able to determine ifhead-mounted unit 602, in general, and displays 220, in particular, areproperly positioned in front of the user's eyes. In other words, theregistration observer 620 may determine if a left display in displaysystem 220 is appropriately aligned with the user's left eye and a rightdisplay in display system 220 is appropriately aligned with the user'sright eye. The registration observer 620 may determine if thehead-mounted unit 602 is properly positioned by determining if thehead-mounted unit 602 is positioned and oriented within a desired rangeof positions and/or orientations relative to the user's eyes.

In at least some embodiments, registration observer 620 may generateuser feedback in the form of alerts, messages, or other content. Suchfeedback may be provided to the user to inform the user of anymisalignment of the head-mounted unit 602, along with optional feedbackon how to correct the misalignment (such as a suggestion to adjust thehead-mounted unit 602 in a particular manner).

Example registration observation and feedback techniques, which may beutilized by registration observer 620, are described in U.S. patentapplication Ser. No. 15/717,747, filed Sep. 27, 2017, which isincorporated by reference herein in its entirety.

Example of an Eye Tracking Module

A detailed block diagram of an example eye tracking module 614 is shownin FIG. 7A. As shown in FIG. 7A, eye tracking module 614 may include avariety of different submodules, may provide a variety of differentoutputs, and may utilize a variety of available data in tracking theuser's eyes. As examples, eye tracking module 614 may utilize availabledata including eye tracking extrinsics and intrinsics, such as thegeometric arrangements of the eye tracking camera 324 relative to thelight sources 326 and the head-mounted-unit 602; assumed eye dimensions704 such as a typical distance of approximately 4.7 mm between a user'scenter of cornea curvature and the average center of rotation of theuser's eye or typical distances between a user's center of rotation andcenter of perspective; and per-user calibration data 706 such as aparticular user's interpupillary distance. Additional examples ofextrinsics, intrinsics, and other information that may be employed bythe eye tracking module 614 are described in U.S. patent applicationSer. No. 15/497,726, filed Apr. 26, 2017, which is incorporated byreference herein in its entirety.

Image preprocessing module 710 may receive images from an eye camerasuch as eye camera 324 and may perform one or more preprocessing (i.e.,conditioning) operations on the received images. As examples, imagepreprocessing module 710 may apply a Gaussian blur to the images, maydown sample the images to a lower resolution, may applying an unsharpmask, may apply an edge sharpening algorithm, or may apply othersuitable filters that assist with the later detection, localization, andlabelling of glints, a pupil, or other features in the images from eyecamera 324. The image preprocessing module 710 may apply a low-passfilter or a morphological filter such as an open filter, which canremove high-frequency noise such as from the pupillary boundary 516 a(see FIG. 5 ), thereby removing noise that can hinder pupil and glintdetermination. The image preprocessing module 710 may outputpreprocessed images to the pupil identification module 712 and to theglint detection and labeling module 714.

Pupil identification module 712 may receive preprocessed images from theimage preprocessing module 710 and may identify regions of those imagesthat include the user's pupil. The pupil identification module 712 may,in some embodiments, determine the coordinates of the position, orcoordinates, of the center, or centroid, of the user's pupil in the eyetracking images from camera 324. In at least some embodiments, pupilidentification module 712 may identify contours in eye tracking images(e.g., contours of pupil iris boundary), identify contour moments (i.e.,centers of mass), apply a starburst pupil detection and/or a canny edgedetection algorithm, reject outliers based on intensity values, identifysub-pixel boundary points, correct for eye-camera distortion (i.e.,distortion in images captured by eye camera 324), apply a random sampleconsensus (RANSAC) iterative algorithm to fit an ellipse to boundariesin the eye tracking images, apply a tracking filter to the images, andidentify sub-pixel image coordinates of the user's pupil centroid. Thepupil identification module 712 may output pupil identification data,which may indicate which regions of the preprocessing images module 712identified as showing the user's pupil, to glint detection and labelingmodule 714. The pupil identification module 712 may provide the 2Dcoordinates of the user's pupil (i.e., the 2D coordinates of thecentroid of the user's pupil) within each eye tracking image to glintdetection module 714. In at least some embodiments, pupil identificationmodule 712 may also provide pupil identification data of the same sortto coordinate system normalization module 718.

Pupil detection techniques, which may be utilized by pupilidentification module 712, are described in U.S. Patent Publication No.2017/0053165, published Feb. 23, 2017 and in U.S. Patent Publication No.2017/0053166, published Feb. 23, 2017, each of which is incorporated byreference herein in its entirety.

Glint detection and labeling module 714 may receive preprocessed imagesfrom module 710 and pupil identification data from module 712. Glintdetection module 714 may use this data to detect and/or identify glints(i.e., reflections off of the user's eye of the light from light sources326) within regions of the preprocessed images that show the user'spupil. As an example, the glint detection module 714 may search forbright regions within the eye tracking image, sometimes referred toherein as “blobs” or local intensity maxima, that are in the vicinity ofthe user's pupil. In at least some embodiments, the glint detectionmodule 714 may rescale (e.g., enlarge) the pupil ellipse to encompassadditional glints. The glint detection module 714 may filter glints bysize and/or by intensity. The glint detection module 714 may alsodetermine the 2D positions of each of the glints within the eye trackingimage. In at least some examples, the glint detection module 714 maydetermine the 2D positions of the glints relative to the user's pupil,which may also be referred to as the pupil-glint vectors. Glintdetection and labeling module 714 may label the glints and output thepreprocessing images with labeled glints to the 3D cornea centerestimation module 716. Glint detection and labeling module 714 may alsopass along data such as preprocessed images from module 710 and pupilidentification data from module 712.

Pupil and glint detection, as performed by modules such as modules 712and 714, can use any suitable techniques. As examples, edge detectioncan be applied to the eye image to identify glints and pupils. Edgedetection can be applied by various edge detectors, edge detectionalgorithms, or filters. For example, a Canny Edge detector can beapplied to the image to detect edges such as in lines of the image.Edges may include points located along a line that correspond to thelocal maximum derivative. For example, the pupillary boundary 516 a (seeFIG. 5 ) can be located using a Canny edge detector. With the locationof the pupil determined, various image processing techniques can be usedto detect the “pose” of the pupil 116. Determining an eye pose of an eyeimage can also be referred to as detecting an eye pose of the eye image.The pose can also be referred to as the gaze, pointing direction, or theorientation of the eye. For example, the pupil may be looking leftwardstowards an object, and the pose of the pupil could be classified as aleftwards pose. Other methods can be used to detect the location of thepupil or glints. For example, a concentric ring can be located in an eyeimage using a Canny Edge detector. As another example, anintegro-differential operator can be used to find the pupillary orlimbus boundaries of the iris. For example, the Daugmanintegro-differential operator, the Hough transform, or other irissegmentation techniques can be used to return a curve that estimates theboundary of the pupil or the iris.

3D cornea center estimation module 716 may receive preprocessed imagesincluding detected glint data and pupil identification data from modules710, 712, 714. 3D cornea center estimation module 716 may use these datato estimate the 3D position of the user's cornea. In some embodiments,the 3D cornea center estimation module 716 may estimate the 3D positionof an eye's center of cornea curvature or a user's corneal sphere, i.e.,the center of an imaginary sphere having a surface portion generallycoextensive with the user's cornea. The 3D cornea center estimationmodule 716 may provide data indicating the estimated 3D coordinates ofthe corneal sphere and/or user's cornea to the coordinate systemnormalization module 718, the optical axis determination module 722,and/or the light-field render controller 618. Further details of theoperation of the 3D cornea center estimation module 716 are providedherein in connection with FIGS. 8A-8E. Techniques for estimating thepositions of eye features such as a cornea or corneal sphere, which maybe utilized by 3D cornea center estimation module 716 and other modulesin the wearable systems of the present disclosure are discussed in U.S.patent application Ser. No. 15/497,726, filed Apr. 26, 2017, which isincorporated by reference herein in its entirety.

Coordinate system normalization module 718 may optionally (as indicatedby its dashed outline) be included in eye tracking module 614.Coordinate system normalization module 718 may receive data indicatingthe estimated 3D coordinates of the center of the user's cornea (and/orthe center of the user's corneal sphere) from the 3D cornea centerestimation module 716 and may also receive data from other modules.Coordinate system normalization module 718 may normalize the eye cameracoordinate system, which may help to compensate for slippages of thewearable device (e.g., slippages of the head-mounted component from itsnormal resting position on the user's head, which may be identified byregistration observer 620). Coordinate system normalization module 718may rotate the coordinate system to align the z-axis (i.e., the vergencedepth axis) of the coordinate system with the cornea center (e.g., asindicated by the 3D cornea center estimation module 716) and maytranslate the camera center (i.e., the origin of the coordinate system)to a predetermined distance away from the cornea center such as 30 mm(i.e., module 718 may enlarge or shrink the eye tracking image dependingon whether the eye camera 324 was determined to be nearer or furtherthan the predetermined distance). With this normalization process, theeye tracking module 614 may be able to establish a consistentorientation and distance in the eye tracking data, relativelyindependent of variations of headset positioning on the user's head.Coordinate system normalization module 718 may provide 3D coordinates ofthe center of the cornea (and/or corneal sphere), pupil identificationdata, and preprocessed eye tracking images to the 3D pupil centerlocator module 720. Further details of the operation of the coordinatesystem normalization module 718 are provided herein in connection withFIGS. 9A-9C.

3D pupil center locator module 720 may receive data, in the normalizedor the unnormalized coordinate system, including the 3D coordinates ofthe center of the user's cornea (and/or corneal sphere), pupil locationdata, and preprocessed eye tracking images. 3D pupil center locatormodule 720 may analyze such data to determine the 3D coordinates of thecenter of the user's pupil in the normalized or unnormalized eye cameracoordinate system. The 3D pupil center locator module 720 may determinethe location of the user's pupil in three-dimensions based on the 2Dposition of the pupil centroid (as determined by module 712), the 3Dposition of the cornea center (as determined by module 716), assumed eyedimensions 704 such as the size of the a typical user's corneal sphereand the typical distance from the cornea center to the pupil center, andoptical properties of eyes such as the index of refraction of the cornea(relative to the index of refraction of air) or any combination ofthese. Further details of the operation of the 3D pupil center locatormodule 720 are provided herein in connection with FIGS. 9D-9G.Techniques for estimating the positions of eye features such as a pupil,which may be utilized by 3D pupil center locator module 720 and othermodules in the wearable systems of the present disclosure are discussedin U.S. patent application Ser. No. 15/497,726, filed Apr. 26, 2017,which is incorporated by reference herein in its entirety.

Optical axis determination module 722 may receive data from modules 716and 720 indicating the 3D coordinates of the center of the user's corneaand the user's pupil. Based on such data, the optical axis determinationmodule 722 may identify a vector from the position of the cornea center(i.e., from the center of the corneal sphere) to the center of theuser's pupil, which may define the optical axis of the user's eye.Optical axis determination module 722 may provide outputs specifying theuser's optical axis to modules 724, 728, 730, and 732, as examples.

Center of rotation (CoR) estimation module 724 may receive data frommodule 722 including parameters of the optical axis of the user's eye(i.e., data indicating the direction of the optical axis in a coordinatesystem with a known relation to the head-mounted unit 602). CoRestimation module 724 may estimate the center of rotation of a user'seye (i.e., the point around which the user's eye rotates when the usereye rotates left, right, up, and/or down). While eyes may not rotateperfectly around a singular point, assuming a singular point may besufficient. In at least some embodiments, CoR estimation module 724 mayestimate an eye's center of rotation by moving from the center of thepupil (identified by module 720) or the center of curvature of thecornea (as identified by module 716) toward the retina along the opticalaxis (identified by module 722) a particular distance. This particulardistance may be an assumed eye dimension 704. As one example, theparticular distance between the center of curvature of the cornea andthe CoR may be approximately 4.7 mm. This distance may be varied for aparticular user based on any relevant data including the user's age,sex, vision prescription, other relevant characteristics, etc.

In at least some embodiments, the CoR estimation module 724 may refineits estimate of the center of rotation of each of the user's eyes overtime. As an example, as time passes, the user will eventually rotatetheir eyes (to look somewhere else, at something closer, further, orsometime left, right, up, or down) causing a shift in the optical axisof each of their eyes. CoR estimation module 724 may then analyze two(or more) optical axes identified by module 722 and locate the 3D pointof intersection of those optical axes. The CoR estimation module 724 maythen determine the center of rotation lies at that 3D point ofintersection. Such a technique may provide for an estimate of the centerof rotation, with an accuracy that improves over time. Varioustechniques may be employed to increase the accuracy of the CoRestimation module 724 and the determined CoR positions of the left andright eyes. As an example, the CoR estimation module 724 may estimatethe CoR by finding the average point of intersection of optical axesdetermined for various different eye poses over time. As additionalexamples, module 724 may filter or average estimated CoR positions overtime, may calculate a moving average of estimated CoR positions overtime, and/or may apply a Kalman filter and known dynamics of the eyesand eye tracking system to estimate the CoR positions over time. As aspecific example, module 724 may calculate a weighted average ofdetermined points of optical axes intersection and assumed CoR positions(such as 4.7 mm from an eye's center of cornea curvature), such that thedetermined CoR may slowly drift from an assumed CoR position (i.e., 4.7mm behind an eye's center of cornea curvature) to a slightly differentlocation within the user's eye over time as eye tracking data for theuser is obtain and thereby enables per-user refinement of the CoRposition.

Interpupillary distance (IPD) estimation module 726 may receive datafrom CoR estimation module 724 indicating the estimated 3D positions ofthe centers of rotation of the user's left and right eyes. IPDestimation module 726 may then estimate a user's IPD by measuring the 3Ddistance between the centers of rotation of the user's left and righteyes. In general, the distance between the estimated CoR of the user'sleft eye and the estimated CoR of the user's right eye may be roughlyequal to the distance between the centers of a user's pupils, when theuser is looking at optical infinity (i.e., the optical axes of theuser's eyes are substantially parallel to one another), which is thetypical definition of interpupillary distance (IPD). A user's IPD may beused by various components and modules in the wearable system. Asexample, a user's IPD may be provided to registration observer 620 andused in assessing how well the wearable device is aligned with theuser's eyes (e.g., whether the left and right display lenses areproperly spaced in accordance with the user's IPD). As another example,a user's IPD may be provided to vergence depth estimation module 728 andbe used in determining a user's vergence depth. Module 726 may employvarious techniques, such as those discussed in connection with CoRestimation module 724, to increase the accuracy of the estimated IPD. Asexamples, IPD estimation module 724 may apply filtering, averaging overtime, weighted averaging including assumed IPD distances, Kalmanfilters, etc. as part of estimating a user's IPD in an accurate manner.

In some embodiments, IPD estimation module 726 may receive data from 3Dpupil center locator module and/or 3D cornea center estimationmodulation 716 indicating the estimated 3D positions of the user'spupils and/or corneas. IPD estimation module 726 may then estimate auser's IPD by reference to the distances between the pupils and corneas.In general, these distances will vary over time as a user rotates theireyes and changes the depth of their vergence. In some cases, the IPDestimation module 726 may look for the largest measured distance betweenthe pupils and/or corneas, which should occur while the user is lookingnear optical infinity and should generally correspond to the user'sinterpupillary distance. In other cases, the IPD estimation module 726may fit the measured distances between the user's pupils (and/orcorneas) to a mathematical relationship of how a person's interpupildistance changes as a function of their vergence depth. In someembodiments, using these or other similar techniques, the IPD estimationmodule 726 may be able to estimate the user's IPD even without anobservation of the user looking at optical infinity (e.g., byextrapolating out from one or more observations in which the user wasverging at distances closer than optical infinity).

Vergence depth estimation module 728 may receive data from variousmodules and submodules in the eye tracking module 614 (as shown inconnection with FIG. 7A). In particular, vergence depth estimationmodule 728 may employ data indicating estimated 3D positions of pupilcenters (e.g., as provided by module 720 described above), one or moredetermined parameters of optical axes (e.g., as provided by module 722described above), estimated 3D positions of centers of rotation (e.g.,as provided by module 724 described above), estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as provided by module 726 described above), and/or oneor more determined parameters of optical and/or visual axes (e.g., asprovided by module 722 and/or module 730 described below). Vergencedepth estimation module 728 may detect or otherwise obtain a measure ofa user's vergence depth, which may be the distance from the user atwhich the user's eyes are focused. As examples, when the user is lookingat an object three feet in front of them, the user's left and right eyeshave a vergence depth of three feet; and, while when the user is lookingat a distant landscape (i.e., the optical axes of the user's eyes aresubstantially parallel to one another such that the distance between thecenters of the user's pupils may be roughly equal to the distancebetween the centers of rotation of the user's left and right eyes), theuser's left and right eyes have a vergence depth of infinity. In someimplementations, the vergence depth estimation module 728 may utilizedata indicating the estimated centers of the user's pupils (e.g., asprovided by module 720) to determine the 3D distance between theestimated centers of the user's pupils. The vergence depth estimationmodule 728 may obtain a measure of vergence depth by comparing such adetermined 3D distance between pupil centers to estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as indicated by module 726 described above). Inaddition to the 3D distance between pupil centers and estimated IPD, thevergence depth estimation module 728 may utilize known, assumed,estimated, and/or determined geometries to calculate vergence depth. Asan example, module 728 may combine 3D distance between pupil centers,estimated IPD, and 3D CoR positions in a trigonometric calculation toestimate (i.e., determine) a user's vergence depth. Indeed, anevaluation of such a determined 3D distance between pupil centersagainst estimated IPD may serve to indicate a measure of the user'scurrent vergence depth relative to optical infinity. In some examples,the vergence depth estimation module 728 may simply receive or accessdata indicating an estimated 3D distance between the estimated centersof the user's pupils for purposes of obtaining such a measure ofvergence depth. In some embodiments, the vergence depth estimationmodule 728 may estimate vergence depth by comparing a user's left andright optical axis. In particular, vergence depth estimation module 728may estimate vergence depth by locating the distance from a user atwhich the user's left and right optical axes intersect (or whereprojections of the user's left and right optical axes on a plane such asa horizontal plane intersect). Module 728 may utilize a user's IPD inthis calculation, by setting the zero depth to be the depth at which theuser's left and right optical axes are separated by the user's IPD. Inat least some embodiments, vergence depth estimation module 728 maydetermine vergence depth by triangulating eye tracking data togetherwith known or derived spatial relationships.

In some embodiments, vergence depth estimation module 728 may estimate auser's vergence depth based on the intersection of the user's visualaxes (instead of their optical axes), which may provide a more accurateindication of the distance at which the user is focused on. In at leastsome embodiments, eye tracking module 614 may include optical to visualaxis mapping module 730. As discussed in further detail in connectionwith FIG. 10 , a user's optical and visual axis are generally notaligned. A visual axis is the axis along which a person is looking,while an optical axis is defined by the center of that person's lens andpupil, and may go through the center of the person's retina. Inparticular, a user's visual axis is generally defined by the location ofthe user's fovea, which may be offset from the center of a user'sretina, thereby resulting in different optical and visual axis. In atleast some of these embodiments, eye tracking module 614 may includeoptical to visual axis mapping module 730. Optical to visual axismapping module 730 may correct for the differences between a user'soptical and visual axis and provide information on the user's visualaxis to other components in the wearable system, such as vergence depthestimation module 728 and light-field render controller 618. In someexamples, module 730 may use assumed eye dimensions 704 including atypical offset of approximately 5.2° inwards (nasally, towards a user'snose) between an optical axis and a visual axis. In other words, module730 may shift a user's left optical axis (nasally) rightwards by 5.2°towards the nose and a user's right optical axis (nasally) leftwards by5.2° towards the nose in order to estimate the directions of the user'sleft and right optical axes. In other examples, module 730 may utilizeper-user calibration data 706 in mapping optical axes (e.g., asindicated by module 722 described above) to visual axes. As additionalexamples, module 730 may shift a user's optical axes nasally by between4.0° and 6.5°, by between 4.5° and 6.0°, by between 5.0° and 5.4°, etc.,or any ranges formed by any of these values. In some arrangements, themodule 730 may apply a shift based at least in part upon characteristicsof a particular user such as their age, sex, vision prescription, orother relevant characteristics and/or may apply a shift based at leastin part upon a calibration process for a particular user (i.e., todetermine a particular user's optical-visual axis offset). In at leastsome embodiments, module 730 may also shift the origins of the left andright optical axes to correspond with the user's CoP (as determined bymodule 732) instead of the user's CoR.

Optional center of perspective (CoP) estimation module 732, whenprovided, may estimate the location of the user's left and right centersof perspective (CoP). A CoP may be a useful location for the wearablesystem and, in at least some embodiments, is a position just in front ofa pupil. In at least some embodiments, CoP estimation module 732 mayestimate the locations of a user's left and right centers of perspectivebased on the 3D location of a user's pupil center, the 3D location of auser's center of cornea curvature, or such suitable data or anycombination thereof. As an example, a user's CoP may be approximately5.01 mm in front of the center of cornea curvature (i.e., 5.01 mm fromthe corneal sphere center in a direction that is towards the eye'scornea and that is along the optical axis) and may be approximately 2.97mm behind the outer surface of a user's cornea, along the optical orvisual axis. A user's center of perspective may be just in front of thecenter of their pupil. As examples, a user's CoP may be less thanapproximately 2.0 mm from the user's pupil, less than approximately 1.0mm from the user's pupil, or less than approximately 0.5 mm from theuser's pupil or any ranges between any of these values. As anotherexample, the center of perspective may correspond to a location withinthe anterior chamber of the eye. As other examples, the CoP may bebetween 1.0 mm and 2.0 mm, about 1.0 mm, between 0.25 mm and 1.0 mm,between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

The center of perspective described herein (as a potentially desirableposition for a pinhole of a render camera and an anatomical position ina user's eye) may be a position that serves to reduce and/or eliminateundesired parallax shifts. In particular, the optical system of a user'seye is very roughly equivalent to theoretical system formed by a pinholein front of a lens, projecting onto a screen, with the pinhole, lens,and screen roughly corresponding to a user's pupil/iris, lens, andretina, respectively. Moreover, it may be desirable for there to belittle or no parallax shift when two point light sources (or objects) atdifferent distances from the user's eye are rigidly rotated about theopening of the pinhole (e.g., rotated along radii of curvature equal totheir respective distance from the opening of the pinhole). Thus, itwould seem that the CoP should be located at the center of the pupil ofan eye (and such a CoP may be used in some embodiments). However, thehuman eye includes, in addition to the lens and pinhole of the pupil, acornea that imparts additional optical power to light propagating towardthe retina). Thus, the anatomical equivalent of the pinhole in thetheoretical system described in this paragraph may be a region of theuser's eye positioned between the outer surface of the cornea of theuser's eye and the center of the pupil or iris of the user's eye. Forinstance, the anatomical equivalent of the pinhole may correspond to aregion within the anterior chamber of a user's eye. For various reasonsdiscussed herein, it may be desired to set the CoP to such a positionwithin the anterior chamber of the user's eye. The derivation andsignificance of the CoP are discussed in additional detail in Appendix(Parts I and II), which forms part of this application.

As discussed above, eye tracking module 614 may provide data, such asestimated 3D positions of left and right eye centers of rotation (CoR),vergence depth, left and right eye optical axis, 3D positions of auser's eye, 3D positions of a user's left and right centers of corneacurvature, 3D positions of a user's left and right pupil centers, 3Dpositions of a user's left and right center of perspective, a user'sIPD, etc., to other components, such as light-field render controller618 and registration observer 620, in the wearable system. Eye trackingmodule 614 may also include other submodules that detect and generatedata associated with other aspects of a user's eye. As examples, eyetracking module 614 may include a blink detection module that provides aflag or other alert whenever a user blinks and a saccade detectionmodule that provides a flag or other alert whenever a user's eyesaccades (i.e., quickly shifts focus to another point).

Example of a Render Controller

A detailed block diagram of an example light-field render controller 618is shown in FIG. 7B. As shown in FIGS. 6 and 7B, render controller 618may receive eye tracking information from eye tracking module 614 andmay provide outputs to render engine 622, which may generate images tobe displayed for viewing by a user of the wearable system. As examples,render controller 618 may receive a vergence depth, left and right eyecenters of rotation (and/or centers of perspective), and other eye datasuch as blink data, saccade data, etc.

Depth plane selection module 750 may receive vergence depth informationand other eye data and, based on such data, may cause render engine 622to convey content to a user with a particular depth plane (i.e., at aparticular accommodation or focal distance). As discussed in connectionwith FIG. 4 , a wearable system may include a plurality of discretedepth planes formed by a plurality of waveguides, each conveying imageinformation with a varying level of wavefront curvature. In someembodiments, a wearable system may include one or more variable depthplanes, such as an optical element that conveys image information with alevel of wavefront curvature that varies over time. In these and otherembodiments, depth plane selection module 750 may cause render engine622 to convey content to a user at a selected depth (i.e., cause renderengine 622 to direct display 220 to switch depth planes), based in partof the user's vergence depth. In at least some embodiments, depth planeselection module 750 and render engine 622 may render content atdifferent depths and also generate and/or provide depth plane selectiondata to display hardware such as display 220. Display hardware such asdisplay 220 may perform an electrical depth plane switching in responseto depth plane selection data (which may be control signals) generatedby and/or provided by modules such as depth plane selection module 750and render engine 622.

In general, it may be desirable for depth plane selection module 750 toselect a depth plane matching the user's current vergence depth, suchthat the user is provided with accurate accommodation cues. However, itmay also be desirable to switch depth planes in a discreet andunobtrusive manner. As examples, it may be desirable to avoid excessiveswitching between depth planes and/or it may be desire to switch depthplanes at a time when the user is less likely to notice the switch, suchas during a blink or eye saccade.

Hysteresis band crossing detection module 752 may help to avoidexcessive switching between depth planes, particularly when a user'svergence depth fluctuates at the midpoint or transition point betweentwo depth planes. In particular, module 752 may cause depth planeselection module 750 to exhibit hysteresis in its selection of depthplanes. As an example, modules 752 may cause depth plane selectionmodule 750 to switch from a first farther depth plane to a second closerdepth plane only after a user's vergence depth passes a first threshold.Similarly, module 752 may cause depth plane selection module 750 (whichmay in turn direct displays such as display 220) to switch to the firstfarther depth plane only after the user's vergence depth passes a secondthreshold that is farther from the user than the first threshold. In theoverlapping region between the first and second thresholds, module 750may cause depth plane selection module 750 to maintain whichever depthplane is currently select as the selected depth plane, thus avoidingexcessive switching between depth planes.

Ocular event detection module 750 may receive other eye data from theeye tracking module 614 of FIG. 7A and may cause depth plane selectionmodule 750 to delay some depth plane switches until an ocular eventoccurs. As an example, ocular event detection module 750 may cause depthplane selection module 750 to delay a planned depth plane switch until auser blink is detected; may receive data from a blink detectioncomponent in eye tracking module 614 that indicates when the user iscurrently blinking; and, in response, may cause depth plane selectionmodule 750 to execute the planned depth plane switch during the blinkevent (such by causing module 750 to direct display 220 to execute thedepth plane switch during the blink event). In at least someembodiments, the wearable system may be able to shift content onto a newdepth plane during a blink event such that the user is unlikely toperceive the shift. As another example, ocular event detection module750 may delay planned depth plane switches until an eye saccade isdetected. As discussed in connection with eye blinks, such as anarrangement may facilitate the discretely shifting of depth planes.

If desired, depth plane selection module 750 may delay planned depthplane switches only for a limited period of time before executing thedepth plane switch, even in the absence of an ocular event. Similarly,depth plane selection module 750 may execute a depth plane switch whenthe user's vergence depth is substantially outside of acurrently-selected depth plane (i.e., when the user's vergence depth hasexceeded a predetermined threshold beyond the regular threshold for adepth plane switch), even in the absence of an ocular event. Thesearrangements may help ensure that ocular event detection module 754 doesnot indefinitely delay depth plane switches and does not delay depthplane switches when a large accommodation error is present. Furtherdetails of the operation of depth plane selection module 750, and howthe module may time depth plane switches, are provided herein inconnection with FIG. 13 .

Render camera controller 758 may provide information to render engine622 indicating where the user's left and right eyes are. Render engine622 may then generate content by simulating cameras at the positions ofthe user's left and right eyes and generating content based on theperspectives of the simulated cameras. As discussed above, the rendercamera is a simulated camera for use in rendering virtual image contentpossibly from a database of objects in a virtual world. The objects mayhave locations and orientations relative to the user or wearer andpossibly relative to real objects in the environment surrounding theuser or wearer. The render camera may be included in a render engine torender virtual images based on the database of virtual objects to bepresented to the eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a camera (corresponding to the “rendercamera”) having an aperture, lens, and detector viewing the objects inthe virtual world. The virtual images are taken from the perspective ofsuch a camera having a position of the “render camera.” For example, thevirtual images may be rendered as if captured from the perspective of acamera having a specific location with respect to the user's or wearer'seye so as to provide images that appear to be from the perspective ofthe user or wearer. In some implementations, the images are rendered asif captured from the perspective of a camera having an aperture at aspecific location with respect to the user's or wearer's eye (such asthe center of perspective or center of rotation as discussed herein, orelsewhere).

Render camera controller 758 may determine the positions of the left andright cameras based on the left and right eye centers of rotation (CoR),determined by CoR estimation module 724, and/or based on the left andright eye centers of perspective (CoP), determined by CoP estimationmodule 732. In some embodiments, render camera controller 758 may switchbetween the CoR and CoP locations based on various factors. As examples,the render camera controller 758 may, in various modes, register therender camera to the CoR locations at all times, register the rendercamera to the CoP locations at all times, toggle or discretely switchbetween registering the render camera to the CoR locations andregistering the render camera to the CoP locations over time based onvarious factors, or dynamically register the render camera to any of arange of different positions along the optical (or visual) axis betweenthe CoR and CoP locations over time based on various factors. The CoRand CoP positions may optionally pass through smoothing filter 756 (inany of the aforementioned modes for render camera positioning) which mayaverage the CoR and CoP locations over time to reduce noise in thesepositions and prevent jitter in the render simulated render cameras.

In at least some embodiments, the render camera may be simulated as apinhole camera with the pinhole disposed at the position of theestimated CoR or CoP identified by eye tracking module 614. As the CoPis offset from the CoR, the location of the render camera and itspinhole both shift as the user's eye rotates, whenever the rendercamera's position is based on a user's CoP. In contrast, whenever therender camera's position is based on a user's CoR, the location of therender camera's pinhole does not move with eye rotations, although therender camera (which is behind the pinhole) may, in some embodiments,move with eye rotation. In other embodiments where the render camera'sposition is based on a user's CoR, the render camera may not move (i.e.,rotate) with a user's eye.

Example of a Registration Observer

A block diagram of an example registration observer 620 is shown in FIG.7C. As shown in FIGS. 6, 7A, and 7C, registration observer 620 mayreceive eye tracking information from eye tracking module 614 (FIGS. 6and 7A). As examples, registration observer 620 may receive informationon a user's left and right eye centers of rotation (e.g., thethree-dimensional positions of the user's left and right eye centers ofrotations, which may be on a common coordinate system or have a commonframe of reference with the head-mounted display system 600). As otherexamples, registration observer 620 may receive display extrinsics, fittolerances, and an eye-tracking valid indicator. The display extrinsicsmay include information on the display (e.g., display 200 of FIG. 2 )such as the field of view of the display, the size of one or moredisplay surfaces, and the positions of the display surfaces relative tothe head-mounted display system 600. The fit tolerances may includeinformation on display registration volumes, which may indicate how farthe user's left and right eyes may move from nominal positions beforedisplay performance is impacted. In addition, the fit tolerances mayindicate the amount of display performance impact that is expected as afunction of the positions of the user's eyes.

As shown in FIG. 7C, registration observer 620 may include a 3Dpositional fit module 770. The positional fit module 770 may obtain andanalyze various pieces of data including, as examples, a left eye centerof rotation 3D position (e.g., CoR Left), a right eye center of rotation3D position (e.g., CoR Right), display extrinsics, and fit tolerances.The 3D positional fit module 770 may determine how far the user's leftand right eyes are from the respective left and right eye nominalpositions (e.g., may calculate 3D left error and 3D right error) and mayprovide the error distances (e.g., 3D left error and 3D right error) todevice 3D fit module 772.

3D positional fit module 770 may also compare the error distances to thedisplay extrinsics and the fit tolerances to determine if the users eyeare within a nominal volume, a partially-degraded volume (e.g., a volumein which the performance of display 220 is partially degraded), or in afully degraded or nearly fully degraded volume (e.g., a volume in whichdisplay 220 is substantially unable to provide content to the user'seyes). In at least some embodiments, 3D positional fit module 770 or 3Dfit module 772 may provide an output qualitatively describing the fit ofthe HMD on the user, such as the Quality of Fit output shown in FIG. 7C.As an example, module 770 may provide an output indicating whether thecurrent fit of the HMD on the user is good, marginal, or failed. A goodfit may correspond to a fit that enables the user to view at least acertain percentage of the image (such as 90%), a marginal fit may enablethe user to view at least a lower percentage of the image (such as 80%),while a failed fit may be a fit in which only an even lower percentageof the image is visible to the user.

As another example, the 3D positional fit module 770 and/or device 3Dfit module 772 may calculate a visible area metric, which may be apercentage of the overall area (or pixels) of images display by display220 that are visible to the user. Modules 770 and 772 may calculate thevisible area metric by evaluating the positions of the user's left andright eyes (e.g., which may be based on the centers of rotation of theuser's eyes) relative to display 220 and using one or more models (e.g.,a mathematical or geometric model), one or more look-up tables, or othertechniques or combinations of these and other techniques to determinewhat percentage of the images are visible to the user as a function ofthe positions of the user's eyes. Additionally, modules 770 and 772 maydetermine which regions or portions of the images display by display 220are expected to be visible to the user as a function of the positions ofthe user's eyes.

Registration observer 620 may also include a device 3D fit module 772.Module 772 may receive data from 3D positional fit module 770 and mayalso receive an eye tracking valid indicator, which may be provided byeye tracking module 614 and may indicate whether the eye tracking systemis currently tracking the positions of the user's eyes or if eyetracking data is unavailable or in an error condition (e.g., determinedto be no reliable). Device 3D fit module 772 may, if desired, modifyquality of fit data received from 3D positional fit module 770 dependingon the state of the eye tracking valid data. For example, if the datafrom the eye tracking system is indicated to not be available or to havean error, the device 3D fit module 772 may provide a notification thatthere is an error and/or not provide output to the user regarding fitquality or fit errors.

In at least some embodiments, registration observer 620 may providefeedback to users on the quality of fit as well as details of the natureand magnitude of the error. As examples, the head-mounted display systemmay provide feedback to the user during calibration or fitting processes(e.g., as part of a setup procedure) and may provide feedback duringoperation (e.g., if the fit degrades due to slippage, the registrationobserver 620 may prompt the user to readjust the head-mounted displaysystem). In some embodiments, the registration analysis may be performedautomatically (e.g., during use of the head-mounted display system) andthe feedback may be provided without user input. These are merelyillustrative examples.

Example of Locating a User's Cornea with an Eye Tracking System

FIG. 8A is a schematic diagram of an eye showing the eye's cornealsphere. As shown in FIG. 8A, a user's eye 810 may have a cornea 812, apupil 822, and a lens 820. The cornea 812 may have an approximatelyspherical shape, shown by corneal sphere 814. Corneal sphere 814 mayhave a center point 816, also referred to as a corneal center, and aradius 818. The semispherical cornea of a user's eye may curve aroundthe corneal center 816.

FIGS. 8B-8E illustrate an example of locating a user's corneal center816 using 3D cornea center estimation module 716 and eye tracking module614.

As shown in FIG. 8B, 3D cornea center estimation module 716 may receivean eye tracking image 852 that includes a corneal glint 854. The 3Dcornea center estimation module 716 may then simulate, in an eye cameracoordinate system 850, the known 3D positions of the eye camera 324 andlight source 326 (which may be based on data in eye tracking extrinsics& intrinsics database 702, assumed eye dimensions database 704, and/orper-user calibration data 706) in order to cast a ray 856 in the eyecamera coordinate system. In at least some embodiments, the eye cameracoordinate system 850 may have its origin at the 3D position of the eyetracking camera 324.

In FIG. 8C, 3D cornea center estimation module 716 simulates a cornealsphere 814 a (which may be based on assumed eye dimensions from database704) and corneal curvature center 816 a at a first position. The 3Dcornea center estimation module 716 may then check to see whether thecorneal sphere 814 a would properly reflect light from the light source326 to the glint position 854. As shown in FIG. 8C, the first positionis not a match as the ray 860 a does not intersect light source 326.

Similarly in FIG. 8D, 3D cornea center estimation module 716 simulates acorneal sphere 814 b and corneal curvature center 816 b at a secondposition. The 3D cornea center estimation module 716 then checks to seewhether the corneal sphere 814 b properly reflects light from the lightsource 326 to the glint position 854. As shown in FIG. 8D, the secondposition is also not a match.

As shown in FIG. 8E, the 3D cornea center estimation module 716eventually is able to determine the correct position of the cornealsphere is corneal sphere 814 c and corneal curvature center 816 c. The3D cornea center estimation module 716 confirms the illustrated positionis correct by checking that light from source 326 will properly reflectoff of the corneal sphere and be imaged by camera 324 at the correctlocation of glint 854 on image 852. With this arrangement and with theknown 3D positions of the light source 326, the camera 324, and theoptical properties of the camera (focal length, etc.), the 3D corneacenter estimation module 716 can determine the 3D location of thecornea's center of curvature 816 (relative to the wearable system).

The processes described herein in connection with at least FIGS. 8C-8Emay effectively be an iterative, repetitious, or optimization process toidentify the 3D position of the user's cornea center. As such, any of aplurality of techniques (e.g., iterative, optimization techniques, etc.)may be used to efficiently and quickly prune or reduce the search spaceof possible positions. Moreover, in some embodiments, the system mayinclude two, three, four, or more light sources such as light source 326and some of all of these light sources may be disposed at differentpositions, resulting in multiple glints such as glint 854 located atdifferent positions on image 852 and multiple rays such as ray 856having different origins and directions. Such embodiments may enhancethe accuracy of the 3D cornea center estimation module 716, as themodule 716 may seek to identify a cornea position that results in someor all of the glints & rays being properly reflected between theirrespective light sources and their respective positions on image 852. Inother words and in these embodiments, the positions of some or all ofthe light sources may be relied upon in the 3D cornea positiondetermination (e.g., iterative, optimization techniques, etc.) processesof FIGS. 8B-8E.

Example of Normalizing the Coordinate System of Eye Tracking Images

FIGS. 9A-9C illustrate an example normalization of the coordinate systemof eye tracking images, by a component in the wearable system such ascoordinate system normalization module 718 of FIG. 7A. Normalizing thecoordinate system of eye tracking images relative to a user's pupillocation may compensate for slippage of the wearable system relative toa user's face (i.e., headset slippage) and such normalization mayestablish a consistent orientation and distance between eye trackingimages and a user's eyes.

As shown in FIG. 9A, coordinate system normalization module 718 mayreceive estimated 3D coordinates 900 of a user's center of cornealrotation and may receive un-normalized eye tracking images such as image852. Eye tracking image 852 and coordinates 900 may be in anun-normalized coordinate system 850 that is based on the location of eyetracking camera 324, as an example.

As a first normalization step, coordinate system normalization module718 may rotate coordinate system 850 into rotated coordinate system 902,such that the z-axis (i.e., the vergence depth axis) of the coordinatesystem may be aligned with a vector between the origin of the coordinatesystem and cornea center of curvature coordinates 900, as shown in FIG.9B. In particular, coordinate system normalization module 718 may rotateeye tracking image 850 into rotated eye tracking image 904, until thecoordinates 900 of the user's corneal center of curvature are normal tothe plane of the rotated image 904.

As a second normalization step, coordinate system normalization module718 may translate rotated coordinate system 902 into normalizedcoordinate system 910, such that cornea center of curvature coordinates900 are a standard, normalized distance 906 from the origin ofnormalized coordinate system 910, as shown in FIG. 9C. In particular,coordinate system normalization module 718 may translate rotated eyetracking image 904 into normalized eye tracking image 912. In at leastsome embodiments, the standard, normalized distance 906 may beapproximately 30 millimeters. If desired, the second normalization stepmay be performed prior to the first normalization step.

Example of Locating a User's Pupil Centroid with an Eye Tracking System

FIGS. 9D-9G illustrate an example of locating a user's pupil center(i.e., the center of a user's pupil 822 as shown in FIG. 8A) using 3Dpupil center locator module 720 and eye tracking module 614.

As shown in FIG. 9D, 3D pupil center locator module 720 may receive anormalized eye tracking image 912 that includes a pupil centroid 913(i.e., a center of a user's pupil as identified by pupil identificationmodule 712). The 3D pupil center locator module 720 may then simulatethe normalized 3D position 910 of eye camera 324 to cast a ray 914 inthe normalized coordinate system 910, through the pupil centroid 913.

In FIG. 9E, 3D pupil center locator module 720 may simulate a cornealsphere such as corneal sphere 901 having center of curvature 900 basedon data from 3D cornea center estimation module 716 (and as discussed inmore detail in connection with FIGS. 8B-8E). As an example, the cornealsphere 901 may be positioned in the normalized coordinate system 910based on the location of the center of curvature 816 c identified inconnection with FIG. 8E and based on the normalization processes ofFIGS. 9A-9C. Additionally, 3D pupil center locator module 720 mayidentify a first intersection 916 between ray 914 (i.e., a ray betweenthe origin of normalized coordinate system 910 and the normalizedlocation of a user's pupil) and the simulated cornea, as shown in FIG.9E.

As shown in FIG. 9F, 3D pupil center locator module 720 may determinepupil sphere 918 based on corneal sphere 901. Pupil sphere 918 may sharea common center of curvature with corneal sphere 901, but have a smallradius. 3D pupil center locator module 720 may determine a distancebetween cornea center 900 and pupil sphere 918 (i.e., a radius of pupilsphere 918) based on a distance between the corneal center and the pupilcenter. In some embodiments, the distance between a pupil center and acorneal center of curvature may be determined from assumed eyedimensions 704 of FIG. 7A, from eye tracking extrinsics and intrinsicsdatabase 702, and/or from per-user calibration data 706. In otherembodiments, the distance between a pupil center and a corneal center ofcurvature may be determined from per-user calibration data 706 of FIG.7A.

As shown in FIG. 9G, 3D pupil center locator module 720 may locate the3D coordinates of a user's pupil center based on variety of inputs. Asexamples, the 3D pupil center locator module 720 may utilize the 3Dcoordinates and radius of the pupil sphere 918, the 3D coordinates ofthe intersection 916 between a simulated cornea sphere 901 and a ray 914associated with a pupil centroid 913 in a normalized eye tracking image912, information on the index of refraction of a cornea, and otherrelevant information such as the index of refraction of air (which maybe stored in eye tracking extrinsics & intrinsics database 702) todetermine the 3D coordinates of the center of a user's pupil. Inparticular, the 3D pupil center locator module 720 may, in simulation,bend ray 916 into refracted ray 922 based on refraction differencebetween air (at a first index of refraction of approximately 1.00) andcorneal material (at a second index of refraction of approximately1.38). After taking into account refraction caused by the cornea, 3Dpupil center locator module 720 may determine the 3D coordinates of thefirst intersection 920 between refracted ray 922 and pupil sphere 918.3D pupil center locator module 720 may determine that a user's pupilcenter 920 is located at approximately the first intersection 920between refracted ray 922 and pupil sphere 918. With this arrangement,the 3D pupil center locator module 720 can determine the 3D location ofthe pupil center 920 (relative to the wearable system), in thenormalized coordinate system 910. If desired, the wearable system canun-normalize the coordinates of the pupil center 920 into the originaleye camera coordinate system 850. The pupil center 920 may be usedtogether with the corneal curvature center 900 to determine, among otherthings, a user's optical axis using optical axis determination module722 and a user's vergence depth by vergence depth estimation module 728.

Example of Differences Between Optical and Visual Axes

As discussed in connection with optical to visual mapping module 730 ofFIG. 7A, a user's optical and visual axes are generally not aligned, duein part to a user's visual axis being defined by their fovea and thatfoveae are not generally in the center of a person's retina. Thus, whena person desires to concentrate on a particular object, the personaligns their visual axis with that object to ensure that light from theobject falls on their fovea while their optical axis (defined by thecenter of their pupil and center of curvature of their cornea) isactually slightly offset from that object. FIG. 10 is an example of aneye 1000 illustrating the eye's optical axis 1002, the eye's visual axis1004, and the offset between these axes. Additionally, FIG. 10illustrates the eye's pupil center 1006, the eye's center of corneacurvature 1008, and the eye's average center of rotation (CoR) 1010. Inat least some populations, the eye's center of cornea curvature 1008 maylie approximately 4.7 mm in front, as indicated by dimension 1012, ofthe eye's average center of rotation (CoR) 1010. Additionally, the eye'scenter of perspective 1014 may lie approximately 5.01 mm in front of theeye's center of cornea curvature 1008, about 2.97 mm behind the outersurface 1016 of the user's cornea, and/or just in front of the user'spupil center 1006 (e.g., corresponding to a location within the anteriorchamber of eye 1000). As additional examples, dimension 1012 may between3.0 mm and 7.0 mm, between 4.0 and 6.0 mm, between 4.5 and 5.0 mm, orbetween 4.6 and 4.8 mm or any ranges between any values and any valuesin any of these ranges. The eye's center of perspective (CoP) 1014 maybe a useful location for the wearable system as, in at least someembodiments, registering a render camera at the CoP may help to reduceor eliminate parallax artifacts.

FIG. 10 also illustrates such a within a human eye 1000 with which thepinhole of a render camera can be aligned. As shown in FIG. 10 , thepinhole of a render camera may be registered with a location 1014 alongthe optical axis 1002 or visual axis 1004 of the human eye 1000 closerto the outer surface of the cornea than both (a) the center of the pupilor iris 1006 and (b) the center of cornea curvature 1008 of the humaneye 1000. For example, as shown in FIG. 10 , the pinhole of a rendercamera may be registered with a location 1014 along the optical axis1002 of the human eye 1000 that is about 2.97 millimeters rearward fromthe outer surface of the cornea 1016 and about 5.01 millimeters forwardfrom the center of cornea curvature 1008. The location 1014 of thepinhole of the render camera and/or the anatomical region of the humaneye 1000 to which the location 1014 corresponds can be seen asrepresenting the center of perspective of the human eye 1000. Theoptical axis 1002 of the human eye 1000 as shown in FIG. 10 representsthe most direct line through the center of cornea curvature 1008 and thecenter of the pupil or iris 1006. The visual axis 1004 of the human eye1000 differs from the optical axis 1002, as it represents a lineextending from the fovea of the human eye 1000 to the center of thepupil or iris 1006.

Example Processes of Rendering Content and Checking Registration Basedon Eye Tracking

FIG. 11 is a process flow diagram of an example method 1100 for usingeye tracking in rendering content and providing feedback on registrationin a wearable device. The method 1100 may be performed by the wearablesystem described herein. Embodiments of the method 1100 can be used bythe wearable system to render content and provide feedback onregistration (i.e., fit of the wearable device to the user) based ondata from an eye tracking system.

At block 1110, the wearable system may capture images of a user's eye oreyes. The wearable system may capture eye images using one or more eyecameras 324, as shown at least in the example of FIG. 3 . If desired,the wearable system may also include one or more light sources 326configured to shine IR light on a user's eyes and produce correspondingglints in the eye images captured by eye cameras 324. As discussedherein, the glints may be used by an eye tracking module 614 to derivevarious pieces of information about a user's eye including where the eyeis looking.

At block 1120, the wearable system may detect glints and pupils in theeye images captured in block 1110. As an example, block 1120 may includeprocessing the eye images by glint detection & labeling module 714 toidentify the two-dimensional positions of glints in the eye images andprocessing the eye images by pupil identification module 712 to identifythe two-dimensional positions of pupils in the eye images.

At block 1130, the wearable system may estimate the three-dimensionalpositions of a user's left and right corneas relative to the wearablesystem. As an example, the wearable system may estimate the positions ofthe center of curvature of a user's left and right corneas as well asthe distances between those centers of curvature and the user's left andright corneas. Block 1130 may involve 3D cornea center estimation module716 identifying the position of the centers of curvature as describedherein at least in connection with FIGS. 7A and 8A-8E.

At block 1140, the wearable system may estimate the three-dimensionalpositions of a user's left and right pupil centers relative to thewearable system. As an example, the wearable system and 3D pupil centerlocator module 720 in particular, may estimate the positions of theuser's left and right pupil centers as described at least in connectionwith FIGS. 7A and 9D-9G, as part of block 1140.

At block 1150, the wearable system may estimate the three-dimensionalpositions of a user's left and right centers or rotation (CoR) relativeto the wearable system. As an example, the wearable system and CoRestimation module 724 in particular, may estimate the positions of theCoR for the user's left and right eyes as described at least inconnection with FIGS. 7A and 10 . As a particular example, the wearablesystem may find the CoR of an eye by walking back along the optical axisfrom the center of curvature of a cornea towards the retina.

At block 1160, the wearable system may estimate a user's IPD, vergencedepth, center of perspective (CoP), optical axis, visual axis, and otherdesired attributes from eye tracking data. As examples, IPD estimationmodule 726 may estimate a user's IPD by comparing the 3D positions ofthe left and right CoRs, vergence depth estimation module 728 mayestimate a user's depth by finding an intersection (or nearintersection) of the left and right optical axes or an intersection ofthe left and right visual axes, optical axis determination module 722may identify the left and right optical axes over time, optical tovisual axis mapping module 730 may identify the left and right visualaxes over time, and the CoP estimation module 732 may identify the leftand right centers of perspective, as part of block 1160.

At block 1170, the wearable system may render content and may,optionally, provide feedback on registration (i.e., fit of the wearablesystem to the user's head) based in part on the eye tracking dataidentified in blocks 1120-1160. As an example, the wearable system mayidentify a suitable location for a render camera and then generatecontent for a user based on the render camera's location, as discussedin connection with light-field render controller 618, FIG. 7B, andrender engine 622. As another example, the wearable system may determineif it is properly fitted to the user, or has slipped from its properlocation relative to the user, and may provide optional feedback to theuser indicating whether the fit of the device needs adjustment, asdiscussed in connection with registration observer 620. In someembodiments, the wearable system may adjust rendered content based onimproper or less than ideal registration in an attempt to reduce,minimize or compensate for the effects of improper or mis-registration.

Example of a Registration Coordinate System

FIGS. 12A-12B illustrate an example eye position coordinate system,which may be used for defining three-dimensional positions of a user'sleft and right eyes relative to the display of the wearable systemdescribed herein. As examples, the coordinate system may include axis x,y, and z. Axis z of the coordinate system may correspond to depth, suchthe distance between the plane a user's eyes lie in and the plane thatdisplay 220 lies in (e.g., the direction normal to the plane of thefront of a user's face). Axis x of the coordinate system may correspondto a left-right direction, such as the distance between the users leftand right eyes. Axis y of the coordinate system may correspond to anup-down direction, which may be a vertical direction when the user isupright.

FIG. 12A illustrates a side view of a user's eye 1200 and a displaysurface 1202 (which may be a part of display 220 of FIG. 2 ), while FIG.12B illustrates a top down view of the user's eye 1200 and the displaysurface 1202. Display surface 1202 may be located in front of the user'seyes and may output image light to the user's eyes. As an example,display surface 1202 may comprise one or more out-coupling lightelements, active or pixel display elements, and may be part of a stackof waveguides, such as stacked waveguide assembly 480 of FIG. 4 . Insome embodiments, the display surface 1202 may be planar. In some otherembodiments, the display surface 1202 may have other topologies (e.g.,be curved). It will be appreciated that the display surface 1202 may bea physical surface of the display, or simply a plane or other imaginarysurface from which image light is understood to propagate from thedisplay 220 to the user's eyes.

As shown in FIG. 12A, the user's eye 1200 may have an actual position1204 offset from a nominal position 1206 and the display surface 1202may be at position 1214. FIG. 12A also illustrates the corneal apex 1212of the user's eye 1200. The user's line of sight (e.g., their opticaland/or visual axis) may be substantially along the line between theactual position 1204 and the corneal apex 1212. As shown in FIGS. 12Aand 12B, the actual position 1204 may be offset from the nominalposition 1206 by an z-offset 1210, a y-offset 1208, and an x-offset1209. The nominal position 1206 may represent a preferred position(sometimes referred to as a design position, which may be generallycentered within a desired volume) for the user's eye 1200 with respectto the display surface 1202. As the user's eye 1200 moves away from thenominal position 1206, the performance of display surface 1202 may bedegraded, as discussed herein in connection with FIG. 14 for example.

It will be appreciated that a point or volume associated with the user'seye 1200 may be used to represent the position of the user's eye inanalyses of registration herein. The representative point or volume maybe any point or volume associated with the eye 1200, and preferably isconsistently used. For example, the point or volume may be on or in theeye 1200, or may be disposed away from the eye 1200. In someembodiments, the point or volume is the center of rotation of the eye1200. The center of rotation may be determined as described herein andmay have advantages for simplifying the registration analyses, since itis roughly symmetrically disposed on the various axes within the eye1200 and allows a single display registration volume aligned with theoptical axis to be utilized for the analyses.

FIG. 12A also illustrates that the display surface 1202 may be centeredbelow the user's horizon (as seen along the y-axis when the user islooking straight ahead, with their optical axis parallel to the ground)and may be tilted (with respect to the y-axis). In particular, thedisplay surface 1202 may be disposed somewhat below the user's horizonsuch that the user would have to look downward, at approximately theangle 1216, to look at the center of the display surface 1202, when theeye 1200 is at position 1206. This may facilitate a more natural andcomfortable interaction with the display surface 1202, particularly whenviewing content rendered at shorter depths (or distances from the user),as users may be more comfortable viewing content below their horizonthan above their horizon. Additionally, display surface 1202 may betilted, such as at angle 1218 (with respect to the y-axis) such that,when the user is looking at the center of the display surface 1202(e.g., looking slightly below the user's horizon), the display surface1202 is generally perpendicular to the user's line of sight. In at leastsome embodiments, the display surface 1202 may also be shifted left orright (e.g., along the x-axis) relative to the nominal position of theuser's eye. As an example, a left-eye display surface may be shiftedright-wards and a right-eye display surface may be shifted left-wards(e.g., display surfaces 1202 may be shifted towards each other) suchthat the user's lines of sight hits the centers of the display surfaceswhen focused at some distance less than infinity, which may increaseuser comfort during typical usage on the wearable device.

Example Graphs of Rendering Content in Response to User Eye Movements

FIG. 13 includes a set of example graphs 1200 a-1200 j which illustratehow the wearable system may switch depth planes in response to theuser's eye movements. As discussed herein in connection with FIGS. 4 and7 , the wearable system may include multiple depth planes, where thevarious depth planes are configured to present content to a user at adifferent simulated depth or with a different accommodation cue (i.e.,with various levels of wavefront curvature or light ray divergence). Asan example, the wearable system may include a first depth planeconfigured to simulate a first range of depths and a second depth planeconfigured to simulate a second range of depths and, while these tworanges may desirably overlap to facilitate hysteresis in switching, thesecond range of depths may generally extend to greater distances fromthe user. In such embodiments, the wearable system may track a user'svergence depth, saccade movements, and blinks to switch between thefirst and second depth planes in a manner that avoid excessive depthplane switching, excessive accommodation-vergence mismatches, andexcessive periods of accommodation-vergence mismatch and that seek toreduce the visibility of depth plane switches (i.e., by shifting depthplanes during blinks and saccades).

Graph 1200 a illustrates an example of a user's vergence depth overtime. Graph 1200 b illustrates an example of a user's saccade signal orvelocity of eye movements over time.

Graph 1200 c may illustrate vergence depth data generated by eyetracking module 614 and, in particular, data generated by vergence depthestimation module 728. As shown in graphs 1200 c-1200 h, eye trackingdata may be sampled within eye tracking module 614 at a rate ofapproximately 60 Hz. As shown between graphs 1200 b and 1200 c, eyetracking data within eye tracking module 614 may lag behind a user'sactual eye movements by a delay 1202. As an example, at time t₁ a user'svergence depth may cross a hysteresis threshold 1210 a, but the eyetracking module 614 may not recognize the event until time t₂ afterdelay 1202.

Graph 1200 c also illustrates various thresholds 1210 a, 1210 b, 1210 cin a hysteresis band, which may be associated with transitions betweenfirst and second depth planes (i.e., depth planes #1 and #0 in FIG. 13). In some embodiments, the wearable system may try to display contentwith depth plane #1 whenever a user's vergence depth is greater thanthreshold 1210 b and to display content with depth plane #0 whenever auser's vergence depth is less than threshold 1210 b. However, to avoidexcessive switching, the wearable system may implement hysteresis,whereby the wearable system will not switch from depth plane #1 to depthplane #0 until the user's vergence depth crosses outer threshold 1210 c.Similarly, the wearable system may not switch from depth plane #0 todepth plane #1 until the user's vergence depth crosses outer threshold1210 a.

Graph 1200 d illustrates an internal flag that may be generated by depthplane selection module 750, or hysteresis band crossing detection module752, indicating whether the user's vergence depth is in the volumegenerally associated with depth plane #1 or the volume generallyassociated with depth plane #2 (i.e., whether the user's vergence depthis greater or less than threshold 1210 b).

Graph 1200 e illustrates an internal hysteresis band flag that may begenerated by depth plane section module 750, or hysteresis band crossingdetection module 752, indicating whether a user's vergence depth hascross an outer threshold such as threshold 1210 a or 1210 c. Inparticular, graph 1200 e illustrates a flag indicative of whether theuser's vergence depth has completely crossed a hysteresis band and intoa region outside of the active depth plane's volume (i.e., into a regionassociated with a depth plane other than an active depth plane), thuspotentially leading to undesirable accommodation-vergence mismatch(AVM).

Graph 1200 f illustrates an internal AVM flag that may be generated bydepth plane selection module 750, or hysteresis band crossing detectionmodule 752, indicating whether a user's vergence has been in outside ofthe active depth plane's volume for greater than a predetermined time.The AVM flag may therefore identify when the user may have beensubjected to an undesirable accommodation-vergence mismatch for anearly-excessive or excessive period of time. Additionally oralternatively, the internal AVM flag may also indicate whether a user'svergence has gone a predetermined distance beyond the active depthplane's volume, thus creating a potentially-excessiveaccommodation-vergence mismatches. In other words, the AVM flag mayindicate when a user's vergence has exceeded an additional thresholdeven further from threshold 1210 b than thresholds 1210 a and 1210 c.

Graph 1200 g illustrates an internal blink flag that may be generated byocular event detection module 754, which may determine when a user hasor is blinking. As noted herein, it may be desired to switch depthplanes upon user blink, to reduce the likelihood of the user perceivingthe switch in depth planes.

Graph 1200 h illustrates an example output from depth plane selectionmodule 750. In particular, graph 1200 h shows that depth plane selectionmodule 750 may output an instruction to utilize a selected depth plane,which may change over time, to a render engine such as render engine 622(see FIG. 6 ).

Graphs 1200 i and 1200 j illustrate delays that may be present in thewearable system including a delay by render engine 622 to switch depthplanes and a delay by the display 220, which may need to provide lightassociated with a new image frame in a new depth plane to effectuate achange in depth planes.

Reference will now be made to the events illustrated in graphs 1200a-1200 j at various times (t₀-t₁₀).

Sometime around time to, a user's vergence depth may cross threshold1210 a, which may be an outer hysteresis threshold. After a delayassociated with image capture and signal processing, the wearable systemmay generate a signal, as indicated in graph 1200 e, that indicates thatthe user's vergence depth lies within the hysteresis band. In theexample of graph 1200 e, the eye tracking module 614 may present ahysteresis band exceeded flag at approximately time t₁ in connectionwith the user's vergence depth crossing threshold 1210 a.

The user's vergence depth may continue to decrease from time to untilapproximately time t₄ and may thereafter increase.

At time t₁, a user's vergence depth may cross threshold 1210 b, whichmay be a midpoint between two depth planes such as depth planes #1 and#0. After processing delay 1202, eye tracking module 614 may alter aninternal flag indicating that the user's vergence depth has moved from avolume generally associated with depth plane #1 into a volume generallyassociated with depth plane #0, as illustrated in graph 1200 d.

At time t₃, the eye tracking module 614 may determine that the user'svergence depth, as shown in graph 1200 a, has moved entirely through thehysteresis band and cross outer threshold 1210 c. As a result, the eyetracking module 614 may generate a signal, as indicated in graph 1200 e,that indicates that the user's vergence depth lies outside thehysteresis band. In at least some embodiments, the eye tracking module614 may switch between first and second depth planes only when a user'svergence depth is outside of the hysteresis band between those two depthplanes.

In at least some embodiments, the eye tracking module 614 may beconfigured to switch depth planes at time t₃. In particular, the eyetracking module 614 may be configured to switch depth planes based on adetermination that the vergence depth has moved from the volume of thecurrently selected depth plane (depth plane #1 as indicated by graph1200 h) into the volume of another depth plane (depth plane #0) andentirely crossed a hysteresis band. In other words, the eye trackingmodule 614 may implement a depth plane switch whenever the hysteresisband is exceeded (graph 1200 e is high) and an accommodation-vergencemismatch based on time or magnitude of mismatch is detected (graph 1200f is high). In such embodiments, the eye tracking module 614 may providea signal to render engine 622 instructing render engine 622 to switch tothe other depth plane (depth plane #0). In the example of FIG. 13 ,however, the eye tracking module 614 may be configured to delay depthplane switches until at least one other condition has been satisfied.These additional conditions may include, as examples, a blink condition,an accommodation-vergence mismatch timeout condition, and anaccommodation-vergence magnitude condition.

At time t₄ and in the example of FIG. 13 , the eye tracking module 614may be configured to switch depth planes. In particular, eye trackingmodule 614 may determine that the user's vergence has been in the volumeassociated with depth plane #0 for longer than a predetermined thresholdof time (and optionally, also outside of the hysteresis band for thatperiod of time). Examples of predetermined thresholds of time include 5seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, and 90 secondsand any range between any of these values. Upon such a determination,the eye tracking module 614 may generate an AVM flag, as indicated ingraph 1200 f, and direct render engine 622 to switch to depth plane #0,as indicated in graph 1200 h. In some embodiments, the eye trackingmodule 614 may generate an AVM flag and direct render engine 622 toswitch depth planes if the user's vergence depth is detected to be morethan a threshold distance from the current selected depth volume.

At time t₅ and after delay 1204, the render engine 622 may startrendering content at the newly-selected depth plane #0. After a delay1206 associated with rendering and conveying light to a user through thedisplay 220, the display 220 may be fully switched to the newly-selecteddepth plane #0 by time t₆.

Thus, graphs 1200 a-j illustrates, between times t₀ and t₆, how thesystem may respond to a user's changing vergence and may switch depthplanes after the user's vergence has moved away from a prior depthvolume for more than a predetermined period of time. Graphs 1200 a-j,between times t₇ and t₁₀, may illustrate how the system responds to auser's changing vergence and may switch depth planes upon detection ofthe user blinking, which may be prior to the predetermined period oftime.

At time t₇, the eye tracking module 614 may detect that the user'svergence depth has entered the hysteresis region between depth planes #0and #1 (i.e., that the user's vergence depth has crossed outer threshold1210 c). In response, the eye tracking module 614 may alter a hysteresisflag as shown in graph 1200 e.

At time t₈, the eye tracking module 614 may detect that the user'svergence depth has cross threshold 1210 b and moved from the volumegenerally associated with depth plane #0 into the volume generallyassociated with depth plane #1. As such, the eye tracking module 614 mayalter a depth volume flag, as shown in graph 1200 d.

At time t₉, the eye tracking module 614 may detect that the user'svergence depth has crossed threshold 1210 a and moved out of thehysteresis volume into the volume generally associated exclusively withdepth plane #1. In response, the eye tracking module 614 may alter ahysteresis flag as shown in graph 1200 e.

At around time t₁₀, the user may blink and the eye tracking module 614may detect that blink. As one example, ocular event detection module 754may detect a user's blink. In response, the eye tracking module 614 maygenerate a blink flag, as shown in graph 1200 h. In at least someembodiments, the eye tracking module 614 may implement a depth planeswitch whenever the hysteresis band is exceeded (graph 1200 e is high)and a blink is detected (graph 1200 g is high). Thus, the eye trackingmodule 614 may instruct render engine 622 to switch depth planes at timet₁₀.

Example Processes of Calibration of Depth Plane Selection

As discussed herein, a head-mounted display such as display 220 of FIG.2 may include a plurality of depth planes, each of which provides adifferent amount of wavefront divergence to provide a differentaccommodation cue to a user's eyes. The depth planes may be formed fromoptical elements such the waveguides 432 b, 434 b, 436 b, and 440 b ofFIG. 4 , which may be configured to send image information to user'seyes with desired levels of wavefront divergence, as examples.

In at least some embodiments, the wearable system including display 220may be configured to display image content having an accommodation cuebased on the current fixation point or vergence depth of the user's gaze(e.g., to reduce or minimize accommodation-vergence mismatches). Inother words, the wearable system may be configured to identify thevergence depth of the user's gaze (e.g., using vergence depth estimationmodule 728 of FIG. 7A, as an example) and then display image content onthe depth plane that provides an accommodation cue associated with thecurrent vergence depth. Thus, when a user is looking at opticalinfinity, the wearable system may display image content on a first depthplane that provides an accommodation cue of optical infinity. Incontrast, when the user is looking in the near field (e.g., within ameter), the wearable system may display image content on a second depthplane that provides an accommodation cue within or at least closer tothe near field.

As previously noted, vergence depth estimation, which may be performedby vergence depth estimation module 728 of FIG. 7A, may be based in partupon the interpupillary distance (IPD) of the current user. Inparticular and in some embodiments, determining the vergence depth mayinvolve projecting the optical and/or visual axes of the user's left andright eyes (to determine their respective gazes) and determining wherethose axes intersect in space and thus where the user's fixation pointor vergence depth is. Geometrically, the optical and/or visual axes arethe legs of a triangle, where the base of the triangle is the user's IPDand the tip of the triangle is the user's fixation point or vergencedepth. Thus, it should be understood that user's IPD is useful indetermining the vergence depth.

In various embodiments, the wearable system may be calibrated to aparticular main user. The calibration may include various processes andmay include determining how the user's eyes move as the user focuses onobjects at different locations and depths. The calibration may alsoinclude identifying the IPD of the user (e.g., the distance between theuser's pupils when the user is focused on optical infinity). Thecalibration may also include determining the user's pupil distance whenthe user is focused at objects closer than optical infinity, such as anobject in the near field (e.g., less than 2.0 meters) and objects in anintermediate field (e.g., between about 2.0 and 3.0 meters). Based onsuch calibration data, the wearable system may be able to determine thedepth the user is looking at by monitoring the pupil distance of theuser. In other words, when the user's pupil distance is at its maximum(e.g., is equal to or close to the user's IPD), the wearable system maybe able to infer that the user's vergence distance is at or near opticalinfinity. In contrast, when the user's pupil distance is near itsminimum, the wearable system may be able to infer that the user'svergence distance is close to the user, at a distance determined by thecalibration.

In some embodiments, the wearable system may utilize one or morealternative processes for depth plane selection. As one example, thewearable system may implement a content-based switching scheme. It willbe appreciated that virtual content may include information about thelocation in virtual space at which that content should be located. Giventhis location, the virtual content may effectively specify an associatedamount of wavefront divergence. Consequently, rather than determiningthe fixation point of the user's eyes to switch depth planes (e.g., toswitch the amount of wavefront divergence of light for forming thevirtual object), the display system may be configured to switch depthplanes based upon the desired location in virtual space in which toplace virtual content.

In some embodiments, the wearable system may still make a determinationas to whether the user is looking at the virtual content in order toswitch to the depth plane specified for that virtual content. Forexample, the display system may still track a gaze of the user todetermine whether they are looking at a virtual object, and once thatdetermination is made, may use the depth information associated with thevirtual content to determine whether to switch depth planes. As anotherexample, the wearable system may identify the most likely real orvirtual object that the user is looking at based on an assumption thatthe user would look at a particular real or virtual object. For example,the wearable system may present a video to the user on a 2D virtualscreen 1 meter away from the user. While the user could be looking awayfrom the screen at another object, it may be reasonable to assume thatthe user would be looking at the video screen. In some embodiments, thewearable system may be configured to make the assumption that the useris looking at real or virtual content that has movement or visiblechanges, or more movement or changes than other real or virtual content;for example, the wearable system may assign a score to movement or theamount of change in visual appearance by real or virtual content withinthe user's field of view and make the assumption that the user islooking at the real or virtual content with the highest score (e.g., themost movement or visual change, such as a virtual screen displaying avideo).

Another example of an alternative depth plane selection process isdynamic calibration. Dynamic calibration may be beneficial when acurrent user has not (or has not yet) performed a dedicated calibrationprocess. As an example, dynamic calibration may be utilized when a guestuser is wearing the device. In one example of a dynamic calibrationsystem, the wearable system may collect eye tracking data in order toestimate a current user's IPD and may then estimate the user's vergencedepth using the estimated IPD (and their eye gaze directions asdiscussed in connection with module 728 of FIG. 7A). IPD estimation maybe performed by IPD estimation module 726 of FIG. 7A and additionaldetails and example embodiments are discussed herein in connection withmodule 726. Dynamic calibration may occur as a background process,requiring no specific action from users. In addition, dynamiccalibration may continuously obtain samples or images of the user's eyesto further refine IPD estimates. As discussed in further detail below,the wearable system may estimate a user's IPD as the 95^(th) percentile(or other percentile) of all measured IPD values. In other words, thelargest 5% of the measured IPD values may be excluded, and then thelargest remaining measured IPD value may be taken as the user's IPD. AnIPD value calculated in this manner may be referred to herein as IPD_95.

It will be appreciated that the display system may be configured tocontinually monitor IPD. As such, the number of samples or individualIPD measurements used to determine a value associated with a particularpercentile may increase over time and potentially increase the accuracyof the IPD determination. In some embodiments, the IPD value (e.g., IPD95) may be continually or periodically updated. For example, the IPDvalue may be updated after a predefined amount of time has elapsedand/or after a predefined number of individual IPD measurements has beenmade.

In some embodiments, a dynamic calibration process may seek to identifythe user's IPD (e.g., the maximum pupil distance of the user, such aswhen the user is looking at optical infinity). In such embodiments, thewearable system may be able to calibrate depth plane selection based onthe IPD alone.

As a particular example, it has been determined that if a user's pupildistance is reduced from their maximum IPD by 0.6 mm, the user is likelyfocused at a depth of approximately 78 mm. 78 mm may correspond to aswitch point between depth planes in some embodiments disclosed herein(e.g., the system may prefer to utilize a first depth plane when theuser is focused at less than 78 mm and utilize a second depth plane whenthe user is focused at greater than 78 mm). In embodiments with a switchpoint that occurs at a different depth of focus, the associated pupildistance reduction relative to their maximum IPD will change in relationto the change in the switch point (e.g., from 78 mm to whatever theswitch point is in such embodiments).

In some instances, the wearable system may refine its calculation of theuser's vergence depth by considering not just the difference between theuser's current pupil distance and their maximum IPD, but also byconsidering how such relationships change in relation to the user'smaximum IPD. In particular, the 0.6 mm number discussed above may be anaverage number that applies to a population as a whole and that takesinto account various biases in the wearable system (e.g., on average, auser having a current pupil distance 0.6 mm less than their maximum IPDmay be verging at a distance of 78 mm). However, the actual IPDdifference (between maximum and current) associated with a vergencedistance of 78 mm (or other switch point distance as discussed in thepreceding paragraph) may be larger or greater than 0.6 mm and may be afunction of the user's anatomical IPD. As particular examples, a personwith an IPD of 54 mm may have a vergence distance of 78 mm when theircurrent IPD is 0.73 mm less than their max IPD (e.g., their IPD whenlooking at distances of at least 10 meters), a person with an IPD of 64mm may have a vergence distance of 78 mm when their current IPD is 0.83mm less than their max IPD, and a person with an IPD of 72 mm may have avergence distance of 78 mm when their current IPD is 0.93 mm less thantheir max IPD. These numbers may differ from the 0.6 mm number for avariety of reasons including, but not limited to, the 0.6 mm number doesnot distinguish between users with different IPDs and the 0.6 mm numbermay be in reference to an IPD_95 value (e.g., may be in reference to anIPD value that is actually slightly lower than the user's anatomical IPDwhen looking at optical infinity). In some embodiments, the 0.6 mmnumber may vary depending upon the user's max IPD. For example, apredefined number of the deviations from the 0.6 mm number may beavailable and may be associated with different ranges of max IPD values.

Using such relations, the wearable system may be able to determine theuser's current vergence depth by comparing their maximum IPD with acurrent interpupillary distance (which may be reduced according to oneor more mathematical functions as their vergence distance decreases).Determining the user's maximum IPD may involve, as an example,collecting a data on the user's IPD over time and identifying themaximum IPD within the collected data. In other embodiments, thewearable system may use heuristics or other processes to determine auser's maximum IPD even when the user does not look to optical infinity.In particular, the wearable system may extrapolate the user's maximumIPD from a plurality of pupil distances associated with closer vergencedepth. The wearable system may also encourage the user to look atoptical infinity by presenting virtual content at optical infinity andasking the user to focus their attention on the virtual content.

FIG. 14 is a process flow diagram of an example method 1400 for depthplane selection using an existing calibration, a content-based switchingscheme, and/or a dynamic calibration. The method 1400 may be performedby the wearable system described herein. Embodiments of the method 1400can be used by the wearable system to render content on a depth planethat generally reduces or minimizes any vergence-accommodationmismatches, which could otherwise lead to user discomfort and fatigue.

At block 1402, the wearable system may determine that it is not beingworn by a user. The wearable system may determine that it is not beingworn using one or more sensors such as eye tracking systems like eyecameras 324. As an example, the wearable system may determine that it isnot being worn by a user based on a determination there are no eyes ineye tracking images captured by eye cameras 324. In particular, thewearable system may determine, after failing to detect eyes in the eyetracking images for at least a given period of time (e.g., apredetermined period of time, a dynamically determined period of time,etc.), that the wearable system is not being worn by a user.

Upon detection of the one or more of the user's eyes in eye trackingimages such as those captured by cameras 324, the method 1400 may moveto block 1404. In block 1404, the wearable system may determine that itis being worn. In some embodiments, some or all of the determinationsassociated with block 1402 and/or block 1406 may be made based at leastin part on data from one or more other sensors of the wearable system,such as IMUs, accelerometers, gyroscopes, proximity sensors, touchsensors, and the like. For example, in these embodiments, the wearablesystem may monitor data from one or more IMUs, accelerometers, and/orgyroscopes for an indication that the wearable system has been placed onor removed from the user's head, may monitor data from one or moreproximity sensors and/or touch sensors to detect the physical presenceof the user, or both. For example, the wearable system may compare datareceived from one or more of these sensors, and the wearable system mayhave threshold values associated with the respective sensors. In oneexample, the wearable system may then determine that the device has beenremoved from the user's head based upon values from one or moreproximity sensors meeting or exceeding a threshold value, optionally inconjunction with data from IMUs, accelerometers, and/or gyroscopesindicating sufficient movement (e.g., exceeding a threshold value) tosupport the conclusion that the device has been removed.

At block 1406, the wearable system may attempt to identify the currentuser by performing an identification process. As one example, thewearable system may estimate the IPD of the current user to determine ifthe current user's IPD matches a calibrated user's IPD (e.g., if the twoIPDs are within some threshold of each other). In some embodiments, thewearable system may determine that the current user is a calibrated userif the calibrated IPD and the current user's IPD are within a thresholdof, e.g., 0.5 mm, 1.0 mm, 1.5 mm, or 2.0 mm, of the calibrated user'sIPD. In general, larger thresholds may facilitate faster determinationsand help to ensure that calibrated users are identified and theircalibration parameters used; for example, larger thresholds are biasedtowards finding that the current user is the calibrated user. In atleast some embodiments, the wearable system may be able to determine acurrent user's IPD with relatively high accuracy (e.g., 95%) within arelatively short time frame (e.g., 5-7 seconds of eye tracking data,which may correspond to between about 150 and 200 frames of eye trackingimages).

If there is a match between the IPD of the current user and the IPD ofthe calibrated user, the wearable system may assume that the currentuser is the calibrated user and load an existing calibration at block1408. The existing calibration may be calibration parameters or datagenerated during a calibration process with the calibrated user wearingthe wearable system. In the event that the current user is not actuallythe calibrated user, but merely has a similar IPD, the calibration dataloaded at block 1408 may be an acceptable calibration that providesreasonable performance for the current user, while allowing the currentuser to use the wearable system without performing a more detailedcalibration.

In some embodiments, the wearable system may identify the current userusing measures other than (or in addition to) the user's IPD. Asexamples, the wearable system may ask the user for a username and/orpassword, the wearable system may perform iris scanning (e.g., comparinga current image of the user's iris with a reference image to determinewhether there is a match, with a match being interpreted to mean thatthe current user is a calibrated user), voice recognition (e.g., bycomparing a current sample of the user's voice with a reference voicefile, with a match being interpreted to mean that the current user is acalibrated user), or some combination of these and other authenticationor identification techniques. In some embodiments, two or moreidentification processes may be performed to increase the accuracy ofthe determination of whether or not the current user is a calibrateduser. For example, the current user may be assumed to be a calibrateduser, but may be determined to not be a calibrated user if all of theperformed identification processes failed to identify the current useras a calibrated user. As another example, the results from the variousperformed identification processes may be aggregated into a combinedscore and the current user may be assumed to be a calibrated user unlessthe combined score exceeds a predetermined threshold value.

In some embodiments, if the display system includes multiple calibrationfiles for multiple users, then additional criteria may be needed toselect the appropriate calibration file. For example, the user may beprompted to select the appropriate calibration file, and/or multipleones of the identification schemes disclosed herein may be utilized.

In some embodiments, multiple identification schemes may be utilized toincrease the accuracy of user identification. For example, it will beappreciated that IPD is a relatively coarse identification criteria. Insome embodiments, IPD may be used as a first criteria to identifywhether the current user is likely a calibrated user, and then a moreprecise or accurate identification scheme may be utilized (e.g., irisscanning). Such a multistep identification scheme advantageously mayconserve processing resources since the more accurate identificationscheme may be more resource intensive. Consequently, processingresources may be saved by delaying use of the more accurateresource-intensive identification scheme until the IPD determinationindicates that a calibrated user is present.

With continued reference to FIG. 14 , if the current user's IPD does notmatch the calibrated user's IPD, then the wearable may perform a dynamiccalibration in block 1410. A dynamic calibration may, as discussedherein, involve monitoring eye tracking data to estimate how a user'sinterpupil distance changes as a function of their vergence distance. Asone example, the dynamic calibration may involve estimating a user'smaximum IPD and then using the maximum IPD together with a currentinterpupil distance to estimate a current vergence distance.

In block 1412, the wearable system may implement content-basedswitching. With content-based switching, depth plane selection is basedon the depth of whatever virtual content the user is determined to belooking at (e.g., the most important or interesting content beingdisplayed, which may be identified by content creators; based on a gazeof the user's eyes, etc.). In at least some embodiments, block 1412 maybe performed whenever selected by content creators or other designers ofthe wearable system and regardless of whether or not the current user isa calibrated user. In various embodiments, content-based switching inblock 1412 may be performed when there is no calibrated user. In suchembodiments, block 1406 may be skipped, if desired. In addition, in someembodiments, the content-based switching of block 1412 may be performedwhether or not block 1408 and/or 1410 (and blocks related to theseblocks) are performed or available to the display system; for example,in some embodiments, the display system may perform only block 1412 fordetermining depth plane switching.

As described herein, each virtual object may be associated with locationinformation, such as three-dimensional location information. Thewearable system may present each virtual object to the user based on thelocation information. For example, the location information for aparticular virtual object may indicate X, Y, and Z coordinates at whichthe object is to be presented (e.g., a center or centroid of the objectmay be presented at the coordinates). Thus, the wearable system mayobtain information indicating a depth plane at which each virtual objectis to be presented.

As will be described in more detail below with respect to FIGS. 16A-18 ,the wearable system may assign a respective volume of space (alsoreferred to herein as a “zone” or marker) as surrounding each object.These volumes of space may preferably not overlap. The wearable systemmay identify a gaze of the user, and identify a zone which includes theuser's gaze. For example, the gaze may indicate a three-dimensionallocation at which the user is fixating (e.g., an approximatethree-dimensional location). The wearable system may then causepresentation of virtual content at a depth plane associated with thevirtual object included in the identified zone. Thus, in someembodiments, once the display system makes a determination as to whichobject the user is looking at, switching between depth planes may occurbased on the location or depth plane associated with virtual objectrather than the fixation point of the user's eyes.

In block 1414, the wearable system may perform depth plane switching.The depth plane switching of block 1414 may be performed with theconfiguration parameters loaded or generated in blocks 1408, 1410, or1412. In particular, if the current user is a calibrated user, thenblock 1414 may perform depth plane switching according to theconfiguration parameters generated during calibration with that user. Ifthe current user is not identified as a calibration user and dynamiccalibration was performed in block 1410, then block 1412 may involvedepth plane switching according to calibration parameters generated inblock 1410 as part of dynamic calibration. In at least some embodiments,the calibration parameters generated at block 1410 may be a user's IPDand the depth plane switching in block 1414 may be based on the user'sIPD. If the wearable system is implementing content-based switching inblock 1412 (which may occur when there is no calibrated user and/or whena content creator or user prefers to utilize content-based switching),the depth plane switching may be performed according to the depth of thecontent the user is assumed to be looking at.

With continued reference to FIG. 14 , it will be appreciated that thedisplay system may be configured to continually confirm the identity ofthe user. For example, after block 1414, the display system may beconfigured to return to block 1406 to identify the user. In someinstances, the display system may lose track of a calibrated user thatwas previously detected at block 1406 to be wearing the display device.This may occur, for example, because the calibrated user has taken offthe display device, or may be due to latency issues, or sensing errorsthat erroneously indicate that the display device is no longer beingworn by the user, even where the calibrated user is still wearing thedisplay device. In some embodiments, even when the display devicedetects that the calibrated user is no longer wearing the displaydevice, the display system may continue to use the calibrated user'scalibration profile for a predetermined amount of time or for apredetermined number of frames before switching to the content-baseddepth plane switching scheme or the dynamic calibration scheme of blocks1412 and 1410, respectively. As discussed herein, the display system maybe configured to continuously perform block 1406 and may continue to notdetect the calibrated user. In response to determining that thecalibrated user has gone undetected for the predetermined amount of timeor the predetermined number of frames, then the system may switch to thecontent-based depth plane switching scheme or the dynamic calibrationscheme of blocks 1412 and 1410, respectively. Advantageously, becausethe calibrated user is typically the most likely user of the displaydevice, by continuing to use the calibrated user's calibration profile,the calibrated user may not experience a significant degradation in userexperience in the event that the system erroneously fails to detect thecalibrated user.

Another example of a method for depth plane selection is shown in FIG.15 . FIG. 15 is a process flow diagram of an example method 1500 fordepth plane selection based on a user's interpupillary distance. Themethod 1500 may be performed by the wearable system described herein.Embodiments of the method 1500 can be used by the wearable system torender content on a depth plane that generally reduces or minimizes anyvergence-accommodation mismatches, which could otherwise lead to userdiscomfort and fatigue.

At block 1502, the wearable system may determine that it is not beingworn by a user. As an example, the wearable system may determine it isnot being worn after failing to detect a user's eyes via eye trackingsystems for more than a threshold period of time (such as 5 seconds, 10seconds, 20 seconds, etc.).

At block 1504, the wearable system may determine that is it being wornby a user. The wearable system may determine it is being worn using oneor more sensors. As an example, the wearable system may include aproximity sensor or a touch sensor that is triggered when the wearablesystem is placed on a user's head and may determine the system is beingworn based on signals from such sensors. As another example, thewearable system may determine it is worn after identifying the presenceof a user's eyes in eye tracking images. In some instances, the wearablesystem may be able to distinguish between 1) failure to detect a user'seyes because the user has shut their eyes and 2) failure to detect auser's eyes because the user has removed the wearable system. Thus, whenthe wearable system detects a user's closed eyes, the wearable systemmay decide the device is being worn. In some embodiments, some or all ofthe determinations associated with block 1502 and/or block 1504 may besimilar to or substantially the same as those associated with block 1402and/or block 1404 as described above with reference to FIG. 14 ,respectively.

At block 1506, the wearable system may check to see if it has beenpreviously calibrated to any users. Block 1506 may involve, in variousembodiments, obtaining the IPD of the calibrated user(s), as part ofidentifying whether or not the current user is the calibrated user (orone of the calibrated users, if there are multiple calibrated users). Insome embodiments, the wearable system may implement content-basedswitching, as discussed herein, and may activate content-based switchingat block 1516 if the wearable system has not been previously calibratedto any users (or if any such calibrations have been deleted or removedfrom the wearable system). The wearable system may also implementcontent-based switching for specific content and/or upon request byusers or content creators. In other words, content creators or users maybe able to specify or request the use of content-based switching, evenwhen the wearable system has been previously calibrated and even whenthe wearable system has been calibrated for the current user.

At block 1508, the wearable system may estimate a current user's IPDand, at block 1510, the wearable system may accumulate eye tracking dataon the current user's IPD. The IPD data accumulated in block 1510 may bemeasurements of the user's interpupil distance over time, measurementsof the user's left and right centers of rotation of time, an indicationof the maximum interpupil distance measured over time, or any otherrelevant data. In at least some embodiments, the wearable system may beable to determine a current user's IPD in block 1508 with relativelyhigh accuracy (e.g., 95%) within a relatively short time frame (e.g.,5-7 seconds of eye tracking data, which may correspond to between about150 and 200 frames of eye tracking images).

In some embodiments, the wearable system may estimate a user's IPD as aparticular percentile (e.g., the 95^(th) percentile) of all IPD valuescollected (e.g., during dynamic calibration at block 1518 and/or duringthe accumulation of IPD data at block 1510). For example, for the95^(th) percentile, the largest 5% of the measured IPD values may beexcluded, and then the largest remaining measured IPD value may be takenas the user's IPD. The IPD value calculated in this manner may bereferred to herein as IPD_95. One benefit of calculating a user's IPDvalue in this manner is that outlying values, which may be greater thanthe user's anatomical IPD, can be excluded. Without being limited bytheory, it is believed that with there is a sufficient number ofmeasured IPD values near or at the user's anatomical IPD that the IPD_95value to accurately reflects the user's anatomical IPD. As an example,if roughly 10% of the IPD measures are close to the user's anatomicalIPD, then the IPD_95 value should still reflect the user's anatomicalIPD value even though the largest 5% values were excluded. If desired,other IPD calculations, which may involve excluding a differentpercentage of the largest measured IPD values may be used. As examples,an IPD_100 value may be used in which no IPD values are excluded or anIPD_98 value may be used in which only the largest 2% of values areexcluded (which may be preferable in a system with eye tracking systemsthat produce relatively few outlying IPD measurements). As additionalexamples, an IPD_90 value may be used, an IPD_85 value may be used, orother IPD value with a desired percentage of the measured valuesexcluded.

At block 1512, the wearable system may determine if the estimated IPD ofthe current user is within a threshold value of the IPD of thecalibrated user (or of one of the IPDs of one of the calibrated users,if there are multiple such users). The threshold value may be, asexamples, 0.5 mm, 1.0 mm, 1.5 mm, 2.0 mm, or 2.5 mm. As one particularexample, the threshold value may be 1.0 mm, which may be sufficientlylarge that a calibrated user is recognized as a calibrated user quicklyand accurately, while non-calibrated users can generally be identifiedas non-calibrated users. In general, it may be preferable to falselyidentify non-calibrated users as calibrated users, than to risk failingto identify calibrated users as calibrated users. In other words, falsenegatives in user identification for calibration may be worse than falsepositives, as the desire for optimal performance for calibrated usersmay outweigh minor sacrifices for non-calibrated or guest users, and asthe calibrated users may be more likely or more frequent users of agiven display system.

If the current user's IPD is within the threshold of the calibrateduser's IPD, the wearable system may assign the current user as thecalibrated user and may load the associated calibration parameters inblock 1514. As an example, the calibration parameters loaded in block1514 may include the calibrated user's IPD, a visual-optical axis offsetparameter, and other available calibration parameters.

If the current user's IPD is not within the threshold of the calibrateduser's IPD, the wearable system may assume that the current user is aguest user and may perform dynamic calibration in block 1518 (orcontent-based switching in block 1516). At block 1518, the wearablesystem may activate dynamic calibration (e.g., dynamically generatecalibration parameters), which may include an on-the-fly calibrationbased on the estimated IPD of the current user, as discussed herein.

At block 1516, the wearable system may implement content-based switchingunder circumstances as discussed herein, such as when no priorcalibration data is available. At this block, as discussed herein, depthplane selection may be made according to the depth associated withvirtual content being displayed as opposed to tracking the vergencedepth of the user. Content-based depth plane switching is furtherdiscussed regarding FIGS. 16-17B.

In block 1520, the wearable system may perform depth plane switching.The depth plane switching of block 1520 may be performed with theconfiguration parameters loaded or generated in blocks 1514, 1516, or1518. In particular, if the current user is a calibrated user, thenblock 1520 may perform depth plane switching according to theconfiguration parameters generated during calibration with that user. Ifthe current user is not identified as a calibration user and dynamiccalibration was performed in block 1518, then block 1520 may involvedepth plane switching according to calibration parameters generated inblock 1520 as part of dynamic calibration. In at least some instances,the calibration parameters generated at block 1518 may be a user's IPDand the depth plane switching in block 1520 may be based on the user'sIPD. If the wearable system is implementing content-based switching inblock 1516 (which may occur when there is no calibrated user and/or whena content creator or user selects utilization of content-basedswitching), the depth plane switching of block 1520 may be performedaccording to the depth of the content the user is determined to belooking at.

Content-Based Switching

As described herein, the wearable system may present virtual content viaa particular depth plane. As the virtual content is updated, for exampleas virtual objects move and/or are replaced with different virtualobjects, the wearable system may select different depth planes at whichto the present the virtual content. As additional nonlimiting examples,as a user of the wearable system focuses on different virtual objects,or at different locations with the user's field of view, the wearablesystem may select different depth planes at which to present the virtualcontent.

An example scheme to select a depth plane, referred to herein ascontent-based switching, is further discussed below with reference toFIGS. 16A-17B. In content-based switching, the wearable system mayselect a depth plane at which to present virtual content-based on adetermination of which virtual object the user is likely looking. Insome embodiments, this determination may be made depending on whetherthe user is fixating within a particular zone. For example, the wearablesystem may associate a zone with each virtual object to be presented asvirtual content to the user. A zone may be, for example, a volume ofspace surrounding a virtual object. The volume of space may be a sphere,cube, hyperrectangle, pyramid, or any arbitrary three-dimensionalpolygon (e.g., a polyhedron). As will be described, the zones arepreferably non-overlapping and, thus, the space encompassed by each zonemay only be associated with a single virtual object.

In some embodiments, the wearable system may monitor a gaze of the user,for example, to identify locations at which the user is viewing (e.g.,within the user's field of view). For an example gaze, the wearablesystem may identify a zone which includes a location being viewed by theuser. In this example, having identified the zone and having only onevirtual object in that zone, the wearable system may select a depthplane corresponding to the virtual object included in the identifiedzone. The wearable system may then present the virtual content at theselected depth plane, which is the depth plane associated with thevirtual object.

Advantageously, the wearable system may utilize higher latency eyetracking, or otherwise less accurate eye tracking schemes than may berequired if depth plane switching were dependent upon accuratelydetermining the depth of the fixation point. In some embodiments, forscenarios with multiple virtual objects displayed at disparate depths,statistical probability or spatial correlation may be utilized todistinguish what content a user is likely to be observing or directingattention to. Thus, the wearable system may determine a fixation area orfixation volume (e.g., rather than a precise fixation point), and selecta depth plane based on the fixation area or volume. In some embodiments,weighted factors such as a last application utilized, or degree to whichthe user's gaze changed, can adjust the statistical probability and,thus, the size/shape of the zone. In some embodiments, the size/shape ofthe zone is correlated with the statistical probability that a user isfixating within a particular volume; for example, where there is a largeamount of uncertainty (e.g., where the uncertainty exceeds a thresholddue to noise, low tolerances in the eye-tracking system, the trackingparticulars of an application that was most-recently used, the speed ofchanges in the user's gaze, etc.) the size of the zone may increase, andwhere there is less uncertainty, the size of the zone may decrease.

As will be described in FIG. 16A, in some embodiments, the zones mayencompass a respective angular distance (e.g., extending from a user'seyes to an infinite distance from the user). In this example, thewearable system may therefore require accuracy only sufficient to placethe user's gaze within particular portions of an X and Y plane.

In some embodiments, the display system may transition from performingcontent-based depth plane switching to performing depth plane switchingbased on a dynamic calibration. Such a transition may occur, forexample, when data obtained for performing the content-based depth planeswitching scheme is determined to be unreliable, or where content isprovided at a range of different depths spanning multiple depth planes.As noted herein, the amount of calculated uncertainty in thedetermination of the zone in which a user fixating may vary based onvarious factors such noise, tolerances of the eye-tracking system, thetracking particulars of an application that was most-recently used, thespeed of changes in the user's gaze, etc. In some embodiments, thedisplay system may be configured to transition to depth plane switchingbased on dynamic calibration when the uncertainty (e.g., the uncertaintyassociated with determining a position of a user's fixation point)exceeds a threshold value. In some other embodiments, particular virtualcontent may span across multiple depth planes. For content spanningacross a threshold number of depth planes (e.g., three depth planes),the display system may be configured to transition to depth planeswitching based on dynamic calibration.

In some embodiments, the display system may be configured to transitionperforming content-based depth plane switching to performing depth planeswitching based on dynamic calibration in response to determining thatthe uncertainty associated with the depth of virtual content exceeds athreshold value. For example, the display system may determine that thedepth or location information for virtual content that is being madeavailable by a particular application that is running on the displaysystem is relatively unreliable (e.g., the particular applicationindicates that the depth or location of mixed reality virtual contentrelative to the user is static even when the user's position changes orthe user moves beyond a predetermined level of position change ormovement, respectively) and, in turn, may transition to depth planeswitching based on dynamic calibration. Such a transition may facilitatea more comfortable and/or realistic viewing experience.

Although described primarily within the context of depth planeswitching, it is to be understood that one or more of the techniquesdescribed herein with reference to FIG. 14 and/or FIG. 15 may beleveraged in any of a variety of different display systems that arecapable of outputting light to an eye of a user with different amountsof wavefront divergence. For example, in some embodiments, one or moreof the techniques described herein with reference to FIG. 14 and/or FIG.15 may be leveraged in a head-mounted display system that includes oneor more variable focus elements (VFEs). For example, one or more of thelenses 458, 456, 454, 452 (FIG. 4 ) may be VFEs, as discussed herein. Inthese embodiments, the head-mounted display system may control operationof its one or more VFEs in real-time based at least in part on whether awearer of the head-mounted display system is determined to be acalibrated user or a guest user. That is, the head-mounted displaysystem may control or adjust an amount of wavefront divergence withwhich light is output to the wearer (the focal distance of the light)based at least in part on whether a wearer of the head-mounted displaysystem is determined to be a calibrated user or a guest user. Examplesof architectures and control schemes for variable focus eyepieces aredisclosed in U.S. Publication No. 2016/0110920, published Apr. 21, 2016and in U.S. Publication No. 2017/0293145, published Oct. 12, 2017, eachof which is incorporated by reference herein in its entirety. Otherconfigurations are possible.

FIG. 16A illustrates a representation of a user's field of view 1602.The user's field of view 1602 (a three-dimensional frustum in theillustrated embodiment) may include a plurality of depth planes 1604A-Dextending in a z-direction from the user's eyes 1606. The field of view1602 may be separated into different zones 1608A-D. Each zone maytherefore encompass a particular volume of space included in the fieldof view 1602. In this example, each zone encompasses a volume of spacethat preferably includes all of the different depth planes 1604A-D. Forexample, zone 1608A may extend along the z-direction from the user'seyes 1606 to, for example, an infinite distance from the eyes 1606.Along an orthogonal direction, such as the x-direction, zone 1608A mayextend from an extremity of the field of view 1602 until the borderbetween zones 1608A and 1608B.

In the illustrated embodiment, zone 1608B includes tree virtual object1610. Tree virtual object 1610 is illustrated as being presented atdepth plane C 1604C. For example, the tree virtual object 1610 may beassociated with location information (e.g., stored, or otherwiseaccessible to, the wearable system). As described above, this locationinformation may be utilized to identify a three-dimensional location atwhich the tree virtual object 1610 is to be presented. As illustrated,the display system may also display a book as virtual object 1612. Zone1608C includes the book virtual object 1612. Thus, the wearable systemis presenting virtual content comprising the tree virtual object 1610and the book virtual object 1612.

The user's eyes 1606 may move about the field of view 1602, for examplefixating on, or viewing, different locations with the field of view1602. If the user's eyes 1606 fixate on the tree virtual object 1610, oron the depth plane C, the wearable system may determine to select depthplane C 1604C to present the virtual content. Similarly, if the user'seyes 1606 fixate on the book virtual object 1612, more on the depthplane B, the wearable system may determine to select depth plane B 1604Bto present the virtual content. As described in, at least, FIG. 14 , thewearable system may utilize different schemes to cause selection ofdepth planes.

With respect to content-based switching, the wearable system mayidentify a zone which includes a location being fixated upon by theuser's eyes 1606. The wearable system may then select a depth planecorresponding to the virtual object included in the identified zone. Forexample, FIG. 16A illustrates an example fixation point 1614. Thewearable system may determine, based on the user's eyes 1606, that theuser is viewing fixation point 1614. The wearable system may thenidentify that fixation point 1614 is included in zone 1608C. Since thiszone 1608C includes the book virtual object 1612, the wearable systemmay cause presentation of the virtual content at depth plane C 1604C.Without being limited by theory, this is believed to provide acomfortable viewing experience, since the only virtual object that maybe fixate upon in the zone 1608C is the book virtual object 1612 and, assuch, it would be appropriate to switch to the depth plane of that bookvirtual object 1612.

Consequently, if the user adjusts fixation to fixation point 1616 or1618, the wearable system may maintain presentation at depth plane C1604B. However, if the user adjusts fixation to within zone 1608B, thewearable system may select depth plane C 1604C at which the present thevirtual content. While the example of FIG. 16A includes four zones, itshould be understood that fewer zones, or a greater number of zones, maybe utilized. Optionally, the number of zones may be the same as thenumber of virtual objects, with a unique zone for each virtual object,and, in some embodiments, the number of zones may vary dynamically asthe number virtual objects changes (preferably so long as the zones canextend from the front to the back of the display frustum without morethan one object occupying the same zone e.g., without two or moreobjects being roughly in the same line of sight relative of the user).For example, two zones may be utilized in the example of FIG. 16A, whichhas two virtual objects that do not have overlapping lines of sight tothe user. Optionally, the wearable system may dynamically adjust avolume of space encompassed by each zone. For example, as accuracy ofgaze detection increases or decreases the volumes of space may beincreased, decreased, or the zones adjusted in number.

FIG. 16B illustrates an example, shown in perspective view, of thevolume of space encompassed by each zone of FIG. 16A. As illustrated,each zone 1608A, 1608B, 1608C, and 1608D extends from the front of thedisplay frustum 1602 to the back in the z-direction. As illustrated, onthe x-y axis, the zones extend vertically to segment or divide the fieldof view along the x-axis. In some other embodiments, on the x-y axis,the zones may extend horizontally to segment the field of view along they-axis. In such embodiments, the illustrated frustum may be effectivelyrotated 90° on the x-y plane. In yet other embodiments, on the x-y axis,the zones may segment or divide the field of view on both the x and they axes, thereby forming a grid on the x-y axis. In all theseembodiments, each zone still preferably extends from the front to theback of the display frustum 1602 (e.g., from the nearest plane thatdisplay system may display content to the furthest plane).

FIG. 17A illustrates another representation of a user's field of view1602 for content-based switching. In the example of FIG. 17A, as in FIG.16A, the user's field of view 1602 includes depth planes A-D 1604A-D.However, the volume of space associated with each virtual object isdifferent from that of FIGS. 16A-B. For example, the zone 1702 in whichbook virtual object 1612 is located is illustrated as encompassing anenclosed shape, such as a spherical volume of space, a cylindricalvolume of space, a cube, a polyhedron, etc. Preferably, the zone 1702extends, on the z-axis, less than the entire depth of the displayfrustum 1602. Advantageously, such an arrangement of zones allowsdifferentiation between objects that may be in similar lines of sight tothe user.

Optionally, the wearable system may adjust the zones of FIG. 17A basedon the eye tracking accuracy exceeding one or more thresholds. Forexample, the sizes of the zones may decrease as confidence in trackingaccuracy increases. As another example, it will be appreciated that eachvirtual object has an associated zone. The wearable system may adjustthe zones of FIG. 17A to avoid overlapping zones. For example, if a newvirtual object is to be displayed within a zone corresponding to adifferent virtual object, then the zones for one or both virtual objectsmay be decreased in size to avoid overlap. For example, if the bookvirtual object 1612 in FIG. 17A is moved in front of the tree virtualobject 1610 (e.g., in zone 1608A), the wearable system may adjust thezones in FIG. 17A to adhere closer to the virtual objects (e.g., asillustrated in FIG. 17A) to prevent overlap.

With continued reference to FIG. 17A, the tree virtual object 1610 isillustrated as being included in a first zone 1706 and a second zone1708. Optionally, the first zone 1706 may represent a relatively smallvolume of space surrounding the tree virtual object 1610. The secondzone 1708 may represent a current volume of space surrounding the treevirtual object 1610. For example, as virtual objects move around withinthe field of view 1602, the volume of space encompassed by a zone may beadjusted (e.g., in real-time). In this example, the adjustment mayensure that each zone does not overlap with any other zone. Thus, if thebook virtual object 1612 moves closer to the tree virtual object 1610,the wearable system may reduce the volume of space of the zonesurrounding the tree virtual object 1610, the book virtual object 1612,or both. For example, the second zone 1708 may be reduced to be closerin volume to the first zone 1706.

As noted herein, in content-based depth plane switching, the depth planeassociated with a virtual object governs, rather than the fixationpoint. As example of this is illustrated with respect to fixation point1704. As illustrated, the example zone 1702 of the book virtual object1612 may encompass a volume of space which includes a portion defined bydepth plane B 1604B and depth plane 1604C. In some cases, the wearablesystem determines that the user is fixating at point 1704 in depth planeC, and also identifies this fixation point 1704 as being included inzone 1702. Because the depth plane associated with the book virtualobject 1612 governs the depth plane switching, the wearable systemswitches to the depth plane B of the book virtual object 1612 ratherthan the depth plane C of the fixation point 1704. FIG. 17B illustratesan example of a perspective view of the representation of FIG. 17A.

FIG. 18 illustrates a flowchart of an example process to select a depthplane based on content-based switching. For convenience, the process1800 will be described as being performed by a wearable system of one ormore processors (e.g., the wearable system, such as wearable system 200,described above).

At block 1802, the wearable system presents virtual content at aparticular depth plane. As described above, depth planes may be utilizedto provide accommodation cues to a user of the wearable system. Forexample, each depth plane may be associated with an amount of wavefrontdivergence of light presented by the wearable system to the user. Thevirtual content may comprise one or a plurality of virtual objects.

At block 1804, the wearable system determines a fixation point. Thewearable system may utilize sensors, such as cameras, to estimate athree-dimensional location at which the user is fixating. These camerasmay update at a particular rate, such as 30 Hz, 60 Hz, and so on. Thewearable system may determine vectors extending from the user's eyes(e.g., from the center or pupil of the eye), and estimate athree-dimensional location at which the vectors intersect. In someembodiments, the fixation point may be estimated based on IPD, with apredefined change in IPD from a maximum IPD being assumed to correlatewith fixation at a particular depth plane. In addition, optionally inaddition to IPD, the gaze of the user may be determined to furtherlocate the approximate position of the fixation point. This estimatedlocation may have certain error associated with it, and the wearablesystem may therefore determine a volume of space in which the fixationpoint likely lies relative to virtual content.

At block 1806, the wearable system identifies a zone that includes thefixation point. As discussed regarding FIGS. 16A-17B, the wearablesystem may associate a zone with each virtual object. For example, azone may include a particular (e.g., single) virtual object. Preferably,the zones for different virtual objects do not overlap.

At block 1808, the wearable system selects a depth plane associated withvirtual content included in the identified zone. The wearable system mayidentify the virtual content associated with the zone identified inblock 1806. The wearable system may then obtain information indicating adepth plane at which the virtual content is to be presented. Forexample, the virtual content may be associated with locationinformation, indicating a three-dimensional location. The wearablesystem may then identify a depth plane associated with thethree-dimensional location. This depth plane may be selected to presentthe virtual content.

FIG. 19 illustrates a flowchart of an example process to adjust zonesbased on content-based switching. For convenience, the process 1800 willbe described as being performed by a wearable system of one or moreprocessors (e.g., the wearable system, such as wearable system 200,described above).

At block 1902, the wearable system presents virtual objects. Thewearable system may, as described herein, have zones associated withpresented virtual objects. For example, a zone may encompass a volume ofspace and include a particular virtual object (e.g., a single virtualobject). The zones may be any shape or polyhedron, and may extendinfinitely in one or more directions in some embodiments. Examples ofzones are illustrated in FIGS. 16A-17B.

At block 1904, the wearable system adjusts a number of virtual objectsor adjusts locations of the virtual objects. The wearable system maypresent additional virtual objects, for example at block 1902 5 virtualobjects may be presented. At block 1904, 8 virtual objects may bepresented. Optionally, the wearable system may update locations of thevirtual objects. For example, a virtual object presented in block 1902may be a bee. The virtual object may thus travel about the user's fieldof view.

At block 1906, the wearable system updates the zones associated with oneor more virtual objects. With respect to the example of additionalvirtual objects being presented, the virtual objects may become closertogether. Since each zone may only be allowed to include a singlevirtual object, in some embodiments, the increase in the number ofvirtual objects may require an adjustment to the zones. For example,FIG. 16A illustrates two virtual objects. If additional virtual objectsare included, they may be included in a zone in which either of the twovirtual objects are also included. Thus, the wearable system may adjustthe zones—for example, adjust a volume of space assigned to each zone(e.g., reduce the volume of space). In this way, each zone may include asingle virtual object. With respect to the example of virtual objectsmoving, a moving virtual object may get closer to another virtualobject. Thus, the moving virtual object may extend into a zoneassociated with the other virtual object. Similar to the above, thewearable system may adjust the zones to ensure that each virtual objectis included in its own zone.

As an example of updating, the wearable system may associate zones withthe virtual objects identified in block 1902. For example, the zones maybe similar to the zones 1608A-1608D illustrated in FIG. 16A. Thewearable system may adjust the zones to encompass smaller volumes ofspace. For example, the wearable system may update the zones to besimilar to the zones 1702, 1706 illustrated in FIG. 17A.

As another example of updating, the zones may be similar to the zones ofFIG. 17A. With respect to FIG. 17A, if the book virtual object 1612moves closer to the tree virtual object 1610, a zone 1708 surroundingthe tree virtual object 1610 may include the book virtual object 1612.Thus, the wearable system may update the zone 1708 to reduce the volumeof space it encompasses. For example, the zone 1708 may be adjusted tobe zone 1706. As another example, zone 1708 may be adjusted to be closerto zone 1706. As escribed in FIG. 17A, zone 1706 may optionally reflecta minimum zone around the tree virtual object 1610.

FIG. 20 illustrates a flowchart of an example process for operating ahead-mounted display system based on user identity. For convenience, themethod 2000 will be described as being performed by a wearable system ofone or more processors (e.g., the wearable system, such as wearablesystem 200, described above). In various implementations of the method2000, the blocks described below can be performed in any suitable orderor sequence, and blocks can be combined or rearranged, or other blockscan be added. In some implementations, the method 2000 may be performedby a head-mounted system that includes one or more cameras that areconfigured to capture images of one or both eyes of a user and one ormore processors coupled to the one or more cameras. In at least some ofsuch implementations, some or all of the operations of method 2000 maybe performed at least in part by the one or more processors of thesystem.

At blocks 2002 through 2006 (i.e., blocks 2002, 2004, and 2006), themethod 2000 may begin with the wearable system determining whether it isnot being worn by a user. The wearable system may determine that it isnot being worn using one or more sensors such as eye tracking systemslike eye cameras 324. In some embodiments, some or all of the operationsassociated with block 2004 may be similar to or substantially the sameas those associated with one or more of blocks 1402, 1404, 1502, and/or1504 as described above with reference to FIGS. 14 and 15 . In someexamples, the method 2000 may remain in a holding pattern until thewearable system determines, at block 2004, that it is being worn by theuser. In response to a determination, at block 2004, that the wearablesystem is being worn by the user, the method 2000 may proceed to block2008.

At block 2008, the method 2000 may include the wearable systemperforming one or more iris authentication operations. In someembodiments, some or all of the operations associated with block 2008may be similar to or substantially the same as those associated with oneor more of blocks 1406 and/or 1506 as described above with reference toFIGS. 14 and 15 , respectively, for implementations in which irisscanning or authentication is employed. In some examples, one or more ofthe operations associated with block 2008 may include the wearablesystem presenting a virtual target to the user (e.g., by way of one ormore displays), capturing one or more images of one or both of theuser's eyes while the virtual target is being presented to the user(e.g., by way of one or more eye tracking cameras), generating one ormore iris codes or tokens based on the one or more images, evaluatingthe one or more generated iris codes or tokens against a set of one ormore predetermined iris codes or tokens that are associated withregistered users, and generating an authentication result based theevaluation. In at least some of these examples, an iris of each of oneor both of the user's eyes may be shown in some or all of the one ormore images. In some implementations, the virtual target may bepresented with the intention of catching the user's attention orotherwise guiding the user's eye gaze in a certain direction and/ortoward a certain location in three-dimensional space. As such, thevirtual target may, for example, include multi-colored and/or animatedvirtual content, may be presented to the user with along withaccompanying audio, haptic feedback, and/or other stimuli, or acombination thereof. In at least some of these implementations, thevirtual target may be presented to the user along with one or moreprompts instructing or suggesting that the user look at or otherwisefixate on the virtual target. In some embodiments, instead of or inaddition to generating and evaluating one or more iris codes or tokens,one or more of the operations associated with block 2008 may include thewearable system may evaluate at least one of the one or more images withone or more reference images to determine whether there is a match, andgenerating an authentication result based the evaluation. Examples ofiris imaging, analysis, recognition, and authentication operations aredisclosed in U.S. patent application Ser. No. 15/934,941 filed on Mar.23, 2018, published on Sep. 27, 2018 as U.S. Publication No.2018/0276467, U.S. patent application Ser. No. 15/291,929 filed on Oct.12, 2016, published on Apr. 20, 2017 as U.S. Publication No.2017/0109580, U.S. patent application Ser. No. 15/408,197 filed on Jan.17, 2017, published on Jul. 20, 2017 as U.S. Publication No.2017/0206412, and U.S. patent application Ser. No. 15/155,013 filed onMay 14, 2016, published on Dec. 8, 2016 as U.S. Publication No.2016/0358181, each of which is incorporated by reference herein in itsentirety. In some embodiments, at block 2008, the method 2000 mayinclude the wearable system performing one or more of the iris imaging,analysis recognition, and authentication operations described in theaforementioned patent applications. In addition, in someimplementations, one or more of the iris imaging, analysis recognition,and authentication operations described in the aforementioned patentapplications may be performed in association with one or more otherlogical blocks of method 2000, such as one or more of blocks 2010, 2012,and 2018 described below. Other configurations are possible.

At block 2010, the method 2000 may include the wearable systemdetermining whether the authentication results (e.g., as obtainedthrough performing one or more iris authentication operations at block2008) indicate whether the user currently wearing the wearable system isa registered user. In some examples, a registered user may be similar toor substantially the same as a calibrated user (e.g., a user with whicha pre-existing calibration profile is associated). In someimplementations, at block 2010, the method 2000 may further include thewearable system determining whether a calibration profile exists for thecurrent user of the device. In some embodiments, some or all of theoperations associated with block 2004 may be similar to or substantiallythe same as those associated with one or more of blocks 1406 and/or 1506as described above with reference to FIGS. 14 and 15 , respectively.

In response to the wearable system determining, at block 2010, that theuser currently wearing the wearable system is indeed a registered user,the method 2000 may proceed to block 2020. At block 2020, the method2000 may include the wearable system selecting settings associated withthe registered user. In some embodiments, some or all of the operationsassociated with block 2020 may be similar to or substantially the sameas those associated with one or more of blocks 1408 and/or 1514 asdescribed above with reference to FIGS. 14 and 15 , respectively.

On the other hand, the method 2000 may proceed from block 2010 to block2012 in response to the wearable system (i) determining, at block 2010,that the user currently wearing the wearable system is not a registereduser or (ii) failing to determine, at block 2010, that the usercurrently wearing the wearable system is a registered user. At block2012, the method 2000 may include the wearable system determiningwhether a confidence value associated with the authentication resultsexceeds a predetermined threshold value. In some embodiments, at block2012, the method 2000 may include the wearable system generating orotherwise obtaining a confidence value associated with theauthentication results, and comparing the confidence value against apredetermined threshold value. In some implementations, the confidencevalue obtained at block 2012 may indicate of a level of confidence thatthe user currently wearing the wearable system is not a registered user.In some examples, the confidence value may be determined based at leastin part on whether or not any iris codes or tokens were successfullygenerated or otherwise extracted at block 2008. A variety of differentfactors and environmental conditions may impact the wearable system'sability to successfully generate or otherwise extract one or more iriscodes or tokens at block 2008. Examples of such factors and conditionsmay include the fit or registration of the wearable system relative tothe current user, the position and/or orientation of the iris of each ofone or both of the user's eyes relative to the position and/ororientation of the one or more cameras at the time the one or moreimages were captured, lighting conditions, or a combination thereof. Assuch, it follows that such factors and conditions may also impact thedegree of confidence with which the wearable system may able todetermine that the current user is not a registered user.

In response to the wearable system determining, at block 2012, that theconfidence value associated with the authentication results does indeedexceed the predetermined threshold value, the method 2000 may proceed toblock 2014. In some examples, such a determination at block 2012 mayindicate that the wearable system has determined that the user currentlywearing the wearable system is not a registered user with a relativelyhigh degree of confidence. At block 2014, the method 2000 may includethe wearable system presenting a prompt to the user (e.g., by way of oneor more displays) suggesting that the user enroll in iris authenticationand/or participate in one or more eye calibration procedures forgenerating a calibration profile for the user. In some embodiments, sucha prompt may request that the user indicate whether they agreed toproceed with the suggested registration procedures. At block 2016, themethod 2000 may include the wearable system receiving user inputresponsive to the prompt and determining whether the received user inputindicates that the user has agreed to proceed with the suggestedregistration procedures. In some examples, at block 2016, the method2000 may monitor received user input(s) and/or pause until the wearablesystem receives user input responsive to the prompt indicating whetherthe user has agreed to proceed with the suggested registrationprocedures.

In response to the wearable system determining, at block 2016, that theuser input received responsive to the prompt indicates that the user hasindeed agreed to proceed with some or all of the suggested registrationprocedures, the method 2000 may proceed to block 2018. At block 2018,the method 2000 may include the wearable system conducting some or allof the suggested registration procedures in accordance with the userinput received responsive to the prompt. In some embodiments, conductingsuch registration procedures may include the wearable system performingone or more operations associated with iris authentication enrollmentand/or eye calibration. As described in further detail below, in someembodiments, such registration procedures associated with block 2018 mayinclude one or more procedures for generating an eye trackingcalibration profile for the current user, one or more operations forenrolling the current user in iris authentication, a combinationthereof. Upon successful completion of the registration proceduresassociated with block 2018, the method 2000 may proceed to block 2020.

On the other hand, the method 2000 may proceed from block 2016 to block2022 in response to the wearable system (i) determining, at block 2016,that the user input received responsive to the prompt indicates that theuser has not agreed to proceed with some or all of the suggestedregistration procedures or (ii) failing to determine, at block 2016,that the user input received responsive to the prompt indicates that theuser has agreed to proceed with some or all of the suggestedregistration procedures. At block 2022, the method 2000 may include thewearable system selecting default settings. In some embodiments, some orall of the operations associated with block 2022 may be similar to orsubstantially the same as those associated with one or more of blocks1410, 1412, 1508-1512, 1516, and/or 1518 as described above withreference to FIGS. 14 and 15 .

In response to the wearable system determining, at block 2012, that theconfidence value associated with the authentication results does notexceed the predetermined threshold value, the method 2000 may proceed tofrom block 2012 to block 2028. In some examples, such a determination atblock 2012 may indicate that the wearable system has relatively littleconfidence in its determination that the user currently wearing thewearable system is not a registered user. At block 2028, the method 2000may include the wearable system determining whether it has attempted toauthenticate the current user (e.g., by performing one or more irisauthentication operations at block 2008) more than a predeterminedthreshold quantity of times. In response to the wearable systemdetermining, at block 2028, that it has indeed attempted to authenticatethe current user more than the predetermined threshold quantity oftimes, the method 2000 may proceed to block 2022.

In response to the wearable system determining, at block 2028, that ithas not yet attempted to authenticate the current user more than thepredetermined threshold quantity of times, the method 2000 may proceedto blocks 2030 through 2032. At block 2032, the method 2000 may includethe wearable system performing one or more operations to attempt toimprove the accuracy and/or reliability of the one or more irisauthentication operations performed in connection with block 2008, so asto enable determinations regarding the user's identity to be made withan increased degree of confidence. As mentioned above, a variety ofdifferent factors and environmental conditions may impact the wearablesystem's ability to successfully generate or otherwise extract one ormore iris codes or tokens. Through performing the one or more operationsassociated with block 2032, the wearable system may attempt tocounteract or otherwise compensate for one or more such factors andconditions. As such, at block 2032, the method 2000 may include thewearable system performing one or more operations so as to increase thelikelihood of successfully generating or otherwise extracting iris codesor tokens from images of one or both of the user's eyes. In someimplementations, at block 2032, the method 2000 may include the wearablesystem performing one or more fit- or registration-related operations,one or more operations for changing the position of the virtual targetsubsequently presented at block 2008, one or more operations fordetermining whether or not the user's eyes are visible, or a combinationthereof.

In some embodiments, such one or more fit- or registration-relatedoperations may correspond to one or more of those described above withreference to registration observer 620 and/or block 1170 of FIG. 11 .That is, in these embodiments, the wearable system may perform one ormore operations to evaluate and/or attempt to improve the fit orregistration of the wearable system relative to the user. To do so, thewearable system may provide the user with feedback and/or suggestionsfor adjusting the fit, positioning, and/or configuration (e.g., nosepads, head pads, etc.) of the wearable system. Additional examples offit- and registration-related operations are disclosed in U.S.Provisional Application No. 62/644,321 filed on Mar. 16, 2018, U.S.application Ser. No. 16/251,017 filed on Jan. 17, 2019, U.S. ProvisionalApplication No. 62/702,866 filed on Jul. 24, 2018, and InternationalPatent Application No. PCT/US2019/043096 filed on Jul. 23, 2019, each ofwhich is incorporated by reference herein in its entirety. In someembodiments, at block 2032, the method 2000 may include the wearablesystem performing one or more of the iris fit- and registration-relatedoperations described in the aforementioned patent applications. Otherconfigurations are possible.

As mentioned above, in some implementations, the method 2000 may includethe wearable system performing one or more operations for changing theposition of the virtual target subsequently presented at block 2008. Inthese examples, the method 2000 may proceed from block 2032 to block2008 and, in performing one or more iris authentication operations inconnection with block 2008, presenting the virtual target for the userat one or more locations different from the location at which thevirtual target was presented to the user in a previous iteration ofmethod 2000. In some examples, the virtual target may be presented tothe user in a manner so as to increase the chances of capturing head-onimages of the iris of each of one or both of the user's eyes. That is,in such examples, the virtual target may be presented to the user at oneor more locations in three-dimensional space that, in order for the userto look at and/or continue to fixate on, may require the user toreorient one or both of their eyes toward the one or more cameras. Insome embodiments, one or more other characteristics of the virtualtarget (e.g., size, shape, color, etc.) may be altered. As alsomentioned above, in some implementations, at block 2032, the method 2000may include the wearable system performing one or more operations fordetermining whether or not the user's eyes are visible. In some suchimplementations, the method 2000 may proceed to block 2022 in responseto the wearable system (i) determining, at block 2032, that the user'seyes are not visible or (ii) failing to determine, at block 2032, thatthe user's eyes are visible.

Upon reaching and performing the operations associated with block 2020or block 2022, the method 2000 may proceed to block 2024. At block 2024,the method 2000 may include the wearable system operating according tothe settings selected at block 2020 or block 2022. In some embodiments,some or all of the operations associated with block 2024 may be similarto or substantially the same as those associated with one or more ofblocks 1414 and/or 1520 as described above with reference to FIGS. 14and 15 , respectively. In some implementations, in transitioning fromblock 2020 or block 2022 to block 2024, the method 2000 may include thewearable system entering an “unlocked” state or otherwise granting thecurrent user access to certain features. At block 2026, the method 2000may include the wearable system determining whether it is not being wornby a user. In some embodiments, some or all of the operations associatedwith block 2004 may be similar to or substantially the same as thoseassociated with one or more of blocks 1402, 1404, 1502, 1504, and/or2004 as described above with reference to FIGS. 14, 15, and 20 .

In response to a determination, at block 2026, that the wearable systemis being worn by the user, the method 2000 may proceed to block 2024.That is, upon reaching blocks 2024-2026, the wearable system maycontinue to operate according to the settings selected at block 2020 orblock 2022 until the wearable system determines, at block 2026, that itis no longer being worn by the user. In response to a determination, atblock 2026, that the wearable system is no longer being worn by theuser, the method 2000 may transition back to block 2004. In someimplementations, in transitioning from block 2026 to block 2004, themethod 2000 may include the wearable system entering a “locked” state orotherwise restricting access to certain features. In some embodiments,in transitioning from block 2026 to block 2004, the method 2000 mayinclude the wearable system suspend or discontinue performance of one ormore of the operations described above with reference to 2024.

As mentioned above, in some embodiments, at block 2018, the method 2000may include the wearable system conducting registration procedures,which may include performing one or more operations for generating aneye tracking calibration profile for the current user, one or moreoperations for enrolling the current user in iris authentication, acombination thereof. In some implementations, at block 2018, the method2000 may include the wearable system performing a calibration for thecurrent user or otherwise performing one or more operations forgenerating an eye tracking calibration profile for the current user. Thecalibration may include various processes and may include determininghow the user's eyes move as the user focuses on objects at differentlocations and depths. In some embodiments, the wearable system may beconfigured to monitor the gaze of a user's eyes and determine a threedimensional fixation point at which the user is fixating. The fixationpoint may be determined based upon, among other things, the distancebetween the user's eyes and the gaze direction of each eye. It will beappreciated that these variables may be understood to form a trianglewith the fixation point at one corner of the triangle and the eyes atthe other corners. It will also be appreciated that a calibration may beperformed to accurately track the orientations of the user's eyes anddetermine or estimate the gaze of those eyes to determine the fixationpoint. The calibration may also include identifying the IPD of the user(e.g., the distance between the user's pupils when the user is focusedon optical infinity). The calibration may also include determining theuser's pupil distance when the user is focused at objects closer thanoptical infinity, such as an object in the near field (e.g., less than2.0 meters) and objects in an intermediate field (e.g., between about2.0 and 3.0 meters). Based on such calibration data, the wearable systemmay be able to determine the depth the user is looking at by monitoringthe pupil distance of the user. In other words, when the user's pupildistance is at its maximum (e.g., is equal to or close to the user'sIPD), the wearable system may be able to infer that the user's vergencedistance is at or near optical infinity. In contrast, when the user'spupil distance is near its minimum, the wearable system may be able toinfer that the user's vergence distance is close to the user, at adistance determined by the calibration. Thus, after undergoing a fullcalibration, the wearable system may store a calibration file and/orcalibration information for the current user. Further details regardingcalibration and eye tracking may be found in U.S. patent applicationSer. No. 15/993,371 filed on May 30, 2018, published on Dec. 6, 2018, asU.S. Publication No. 2018/0348861, U.S. Provisional Application No.62/714,649 filed on Aug. 3, 2018, U.S. Provisional Application No.62/875,474 filed on Jul. 17, 2019, and U.S. application Ser. No.16/530,904 filed on Aug. 2, 2019, each of which is incorporated byreference herein in its entirety. In some embodiments, at block 2018,the method 2000 may include the wearable system performing one or moreof the calibration and eye tracking operations described in theaforementioned patent applications. Other configurations are possible.

In some implementations, at block 2018, the method 2000 may include thewearable system performing one or more operations for enrolling thecurrent user in iris authentication. In some examples, such operationsmay include the wearable system presenting a virtual target to the user(e.g., by way of one or more displays), capturing one or more images ofone or both of the user's eyes while the virtual target is beingpresented to the user (e.g., by way of one or more eye trackingcameras), generating a representation of an iris of each of one or bothof the user's eyes (e.g., one or more iris codes or tokens) based on theone or more images, and storing the generated representation(s). In someimplementations, instead of or in addition to generating and storing arepresentation of an iris of each of one or both of the user's eyes(e.g., one or more iris codes or tokens), such operations may includestoring at least one of the one or more images as a reference ortemplate image. In some implementations, the wearable system may performone or more of the operations associated with block 2032 when performingone or more operations for enrolling the current user in irisauthentication, such as one or more of those associated with block 2018,so as to increase the likelihood of successfully generating or otherwiseextracting iris codes or tokens from images of one or both of the user'seyes. In some embodiments, the wearable system may perform one or moresuch operations for enrolling the current user in iris authenticationconcurrently with the aforementioned calibration procedure for thecurrent user. In other embodiments, the wearable system may perform oneor more such operations for enrolling the current user in irisauthentication and the aforementioned calibration procedure for thecurrent user at separate times.

In some examples, at block 2018, the method 2000 may include thewearable system storing the calibration file or calibration informationand the generated representation(s) (e.g., one or more iris codes ortokens) in association with one another. In some embodiments, instead ofor in addition to storing the calibration file or calibrationinformation and the generated representation(s) in association with oneanother, at block 2018, the method 2000 may include the wearable systemstoring the calibration file or calibration information and at least oneof the one or more images in association with one another. In someembodiments, some or all of the operations associated with block 2018may be performed independent from the method 2000. For example, thewearable system may perform one or more operations for enrolling a userin iris authentication and/or one or more operations associated forperforming a calibration for a user at the request of the user or whenthe wearable system is initially turned on or set up.

In some embodiments, in response to the wearable system determining, atblock 2012, that the confidence value associated with the authenticationresults does not exceed the predetermined threshold value, the method2000 may proceed to from block 2012 to one or more logical blocks otherthan block 2028. For example, in these embodiments, in response to thewearable system determining, at block 2012, that the confidence valueassociated with the authentication results does not exceed thepredetermined threshold value, the wearable system may present thecurrent user with one or more alternative authentication options. Forinstance, the wearable system may provide the current user with theoption to verify their identity by providing a pin, password, and/orother credentials as input to the wearable system. In some of theseembodiments, the method 2000 may proceed to block 2020 upon successfullyidentifying the current user as a registered user using one or more ofthe aforementioned alternative authentication options. Similarly, insome of these embodiments, the method 2000 may proceed to block 2016upon failing to successfully identify the current user as a registereduser using one or more of the aforementioned alternative authenticationoptions. At such a juncture, the method 2000 may either proceed to block2022 or proceed to one or more logical blocks similar to block 2018 inwhich the wearable system conducts one or more operations forregistering the current user.

In some examples, such one or more operations for registering thecurrent user may include one or more procedures for generating an eyetracking calibration profile for the current user, one or moreoperations for enrolling the current user in iris authentication, one ormore operations for enrolling the current user in one or more of theaforementioned alternative authentication options, or a combinationthereof. In some embodiments, at block 2018, the method 2000 may furtherinclude the wearable system performing one or more operations forenrolling the current user in one or more of the aforementionedalternative authentication options.

In these embodiments, at block 2018, the method 2000 may include thewearable system storing information generated or otherwise obtained bythe wearable system in connection with performing one or more operationsfor enrolling the current user in one or more of the aforementionedalternative authentication options in association with calibration filesor information for the current user, one or more iris representations(e.g., iris codes or tokens) generated for the current user, one or moreimages of one or both of the user's eyes, or a combination thereof.Additional information, such as that which is manually supplied by theuser at block 2018 (e.g., name, email address, interests, userpreferences, etc.), may also be stored in association with one or moreof the aforementioned pieces of information. Other configurations arepossible.

Computer Vision to Detect Objects in Ambient Environment

As discussed above, the display system may be configured to detectobjects in or properties of the environment surrounding the user. Thedetection may be accomplished using a variety of techniques, includingvarious environmental sensors (e.g., cameras, audio sensors, temperaturesensors, etc.), as discussed herein.

In some embodiments, objects present in the environment may be detectedusing computer vision techniques. For example, as disclosed herein, thedisplay system's forward-facing camera may be configured to image theambient environment and the display system may be configured to performimage analysis on the images to determine the presence of objects in theambient environment. The display system may analyze the images acquiredby the outward-facing imaging system to perform scene reconstruction,event detection, video tracking, object recognition, object poseestimation, learning, indexing, motion estimation, or image restoration,etc. As other examples, the display system may be configured to performface and/or eye recognition to determine the presence and location offaces and/or human eyes in the user's field of view. One or morecomputer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

One or more of these computer vision techniques may also be usedtogether with data acquired from other environmental sensors (such as,e.g., microphone) to detect and determine various properties of theobjects detected by the sensors.

As discussed herein, the objects in the ambient environment may bedetected based on one or more criteria. When the display system detectsthe presence or absence of the criteria in the ambient environment usinga computer vision algorithm or using data received from one or moresensor assemblies (which may or may not be part of the display system),the display system may then signal the presence of the object.

Machine Learning

A variety of machine learning algorithms may be used to learn toidentify the presence of objects in the ambient environment. Oncetrained, the machine learning algorithms may be stored by the displaysystem. Some examples of machine learning algorithms may includesupervised or non-supervised machine learning algorithms, includingregression algorithms (such as, for example, Ordinary Least SquaresRegression), instance-based algorithms (such as, for example, LearningVector Quantization), decision tree algorithms (such as, for example,classification and regression trees), Bayesian algorithms (such as, forexample, Naive Bayes), clustering algorithms (such as, for example,k-means clustering), association rule learning algorithms (such as, forexample, a-priori algorithms), artificial neural network algorithms(such as, for example, Perceptron), deep learning algorithms (such as,for example, Deep Boltzmann Machine, or deep neural network),dimensionality reduction algorithms (such as, for example, PrincipalComponent Analysis), ensemble algorithms (such as, for example, StackedGeneralization), and/or other machine learning algorithms. In someembodiments, individual models may be customized for individual datasets. For example, the wearable device may generate or store a basemodel. The base model may be used as a starting point to generateadditional models specific to a data type (e.g., a particular user), adata set (e.g., a set of additional images obtained), conditionalsituations, or other variations. In some embodiments, the display systemmay be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

The criteria for detecting an object may include one or more thresholdconditions. If the analysis of the data acquired by the environmentalsensor indicates that a threshold condition is passed, the displaysystem may provide a signal indicating the detection the presence of theobject in the ambient environment. The threshold condition may involve aquantitative and/or qualitative measure. For example, the thresholdcondition may include a score or a percentage associated with thelikelihood of the reflection and/or object being present in theenvironment. The display system may compare the score calculated fromthe environmental sensor's data with the threshold score. If the scoreis higher than the threshold level, the display system may detect thepresence of the reflection and/or object. In some other embodiments, thedisplay system may signal the presence of the object in the environmentif the score is lower than the threshold. In some embodiments, thethreshold condition may be determined based on the user's emotionalstate and/or the user's interactions with the ambient environment.

In some embodiments, the threshold conditions, the machine learningalgorithms, or the computer vision algorithms may be specialized for aspecific context. For example, in a diagnostic context, the computervision algorithm may be specialized to detect certain responses to thestimulus. As another example, the display system may execute facialrecognition algorithms and/or event tracing algorithms to sense theuser's reaction to a stimulus, as discussed herein.

It will be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. For example,computing systems may include general purpose computers (e.g., servers)programmed with specific computer instructions or special purposecomputers, special purpose circuitry, and so forth. A code module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming language.In some embodiments, particular operations and methods may be performedby circuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

Other Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

What is claimed is:
 1. A system comprising: a display configured topresent image light to a user, the image light comprising virtual imagecontent, wherein presenting the image light comprises presenting theimage light with a wavefront divergence corresponding to a depth planeat a distance from the user; a camera; and one or more processorsconfigured for: obtaining, via the camera, an image of an eye of theuser; determining, based on the image of the eye of the user, whetherthe user corresponds to a registered user; in accordance with adetermination that the user corresponds to the registered user:determining a setting associated with the registered user, the settingcorresponding to a first wavefront divergence; and presenting, to theuser via the display, the image light with the first wavefrontdivergence; and in accordance with a determination that the user doesnot correspond to the registered user: presenting, to the user via thedisplay, the image light with a second wavefront divergence differentfrom the first wavefront divergence.
 2. The system of claim 1, whereindetermining whether the user corresponds to the registered usercomprises performing one or more of an iris scanning operation and aniris authentication operation.
 3. The system of claim 1, wherein thesecond wavefront divergence corresponds to a default setting.
 4. Thesystem of claim 1, wherein the one or more processors are furtherconfigured for, in accordance with the determination that the user doesnot correspond to the registered user: obtaining iris information of theuser; and determining the second wavefront divergence based on the irisinformation.
 5. The system of claim 1, wherein the one or moreprocessors are further configured for: presenting, via the display tothe user, a virtual target, wherein obtaining the image of the eye ofthe user comprises obtaining the image of the eye of the userconcurrently with presenting the virtual target to the user.
 6. Thesystem of claim 1, wherein the one or more processors are furtherconfigured for, in accordance with the determination that the usercorresponds to the registered user, improving an accuracy or areliability of user identification based on the determination.
 7. Amethod comprising: presenting, by a display, image light to a user, theimage light comprising virtual image content, wherein presenting theimage light comprises presenting the image light with a wavefrontdivergence corresponding to a depth plane at a distance from the user;obtaining, via a camera, an image of an eye of the user; in accordancewith a determination that the user corresponds to the registered user:determining a setting associated with the registered user, the settingcorresponding to a first wavefront divergence; and presenting, to theuser via the display, the image light with the first wavefrontdivergence; and in accordance with a determination that the user doesnot correspond to the registered user: presenting, to the user via thedisplay, the image light with a second wavefront divergence differentfrom the first wavefront divergence.
 8. The method of claim 7, whereindetermining whether the user corresponds to the registered usercomprises performing one or more of an iris scanning operation and aniris authentication operation.
 9. The method of claim 7, wherein thesecond wavefront divergence corresponds to a default setting.
 10. Themethod of claim 7, further comprising in accordance with thedetermination that the user does not correspond to the registered user:obtaining iris information of the user; and determining the secondwavefront divergence based on the iris information.
 11. The method ofclaim 7, further comprising: presenting, via the display to the user, avirtual target, wherein obtaining the image of the eye of the usercomprises obtaining the image of the eye of the user concurrently withpresenting the virtual target to the user.
 12. The method of claim 7,further comprising in accordance with the determination that the usercorresponds to the registered user, improving an accuracy or areliability of user identification based on the determination.
 13. Anon-transitory computer-readable medium storing instructions that, whenexecuted by one or more processors, cause the one or more processors toperform a method comprising: presenting, by a display, image light to auser, the image light comprising virtual image content, whereinpresenting the image light comprises presenting the image light with awavefront divergence corresponding to a depth plane at a distance fromthe user; obtaining, via a camera, an image of an eye of the user; inaccordance with a determination that the user corresponds to theregistered user: determining a setting associated with the registereduser, the setting corresponding to a first wavefront divergence; andpresenting, to the user via the display, the image light with the firstwavefront divergence; and in accordance with a determination that theuser does not correspond to the registered user: presenting, to the uservia the display, the image light with a second wavefront divergencedifferent from the first wavefront divergence.
 14. The non-transitorycomputer-readable medium of claim 13, wherein determining whether theuser corresponds to the registered user comprises performing one or moreof an iris scanning operation and an iris authentication operation. 15.The non-transitory computer-readable medium of claim 13, wherein thesecond wavefront divergence corresponds to a default setting.
 16. Thenon-transitory computer-readable medium of claim 13, wherein the methodfurther comprises in accordance with the determination that the userdoes not correspond to the registered user: obtaining iris informationof the user; and determining the second wavefront divergence based onthe iris information.
 17. The non-transitory computer-readable medium ofclaim 13, wherein the method further comprises: presenting, via thedisplay to the user, a virtual target, wherein obtaining the image ofthe eye of the user comprises obtaining the image of the eye of the userconcurrently with presenting the virtual target to the user.
 18. Thenon-transitory computer-readable medium of claim 13, wherein the methodcomprises in accordance with the determination that the user correspondsto the registered user, improving an accuracy or a reliability of useridentification based on the determination.